and keys contour extraction from a MIDI keyboard image
keyboard image parser consists of two modules, with one module
performing the extraction of the contour of a hand situated above a
musical instrument keyboard, and the other module performing the
computation of the geometry of the polygons which outline/represent the
individual keys of a musical instrument keyboard.
This project was ordered by a DAW software
startup as part of a VST
plugin solution aimed at providing a low-cost, image processing-based
solution for extracting information on hand and/or finger gestures over a MIDI
keyboard's keys, and then use the extracted information to send MIDI
signals to a DAW in order to achieve various programmable sound effects
(e.g. alter the tonality of the sound, introduce special effects such as
tremolo or vibrato, sliding notes, etc).
Solutions targeting a similar feature set as the
above-mentioned VST plugin already existed on the market at the time
this project was started, but said solutions are all using various kinds
of touch sensors placed on the keyboard, or they consist of special
touch-sensitive membrane designs, etc. No existing solution on the
market at the time of writing (october 2021) provides a similar feature
set via digital image processing.
The videos below provide a glimpse into some sound effects that can be
produced by a DAW by extracting gesture information from the fingers
and/or hands motion over a MIDI keyboard (note that the DAW plugins utilized
in the videos below require the use of specially crafted MIDI keyboards which
contain touch-position detection via [various types of] hardware sensors).