mask creator?

[GDjkhp]

yow you should add an AI that automatically creates masks from the videos lol, like detects black keys and white keys

[minyor]

hmm, I thought about that too, I think it might be challeging to make it fail proof though..
I may be willing to try do something like this if I have more free time or if there will be more interest for this software, some donations maybe? Idno

[goombapatrol]

So I wrote a script to do exactly this; figured I might as well share in case anybody else would find it useful. syn2midi-maskmaker.bat

It uses ffmpeg to extract frame of video, and imagemagick to quickly generate the BMP.

Takes optional geometry parameters to accommodate for videos having keys lower or higher; I've included in the repo a reference image of what output should look like, and where to position the rectangle. You'll want it between all the keys due to how it works.

Set the fuzz variable to a higher value than 5 if the white notes are darker (5-25 seems to be a good range).

The way it works is by creating boundaries; it stacks left and right-shifted images and fills in the between parts.

2024-10-06_15-13-31.mp4

Which finally looks like this. (shrunk PNG for preview sake; full size BMP is here)

---

Let me know if any videos break it, so I can work on making it even more foolproof.

And thanks again @minyor for writing this cool utility :)

[minyor]

Hello, wow I didn't know we could write such complex bat scripts.. Cool :)

[minyor]

Just a quik thought, I think, to make it even more foolproof, you could use some probability averaging.
For example, try to reduce blue line height to a pixel or two and then repeat your algorithm iterating troughout all of keys vertical range, or to simplify, thoughout bottom halve of a video. Then count numbers of blue islands, selecting lines with max number of islands and max number of occurrences. One of these lines of blue islands I guess should be most probable mask