A full-body keyboard
View a fuller demo and more background on the project at https://youtu.be/h376W93gQq4
Semaphore uses OpenCV and MediaPipe’s Pose detection to perform real-time detection of body landmarks from video input. From there, relative differences are calculated to determine specific positions and translate those into keys and commands sent via keyboard.
The primary input is to “type” letters, digits, and symbols via flag semaphore by extending both arms at various angles. Rather than waiting a set time after every signal, you can jump to repeat the last sent symbol.
SEMAPHORES dictionary in the code for a full set
of angles, which mostly conform to standard US semaphore with some custom additions. Most of the rest of the keyboard is included as other modifier gestures, such as:
shift: open both hands, instead of fists
backspace: both hands over mouth
digitsand other extra symbols: squat while signaling
command: lift left leg to ~horizontal thigh
control: lift right leg to ~horizontal thigh
arrow left/right/up/down: cross arms and raise each straight leg
- repeat previous letter/command: jump
Running on latest MacOS from Terminal, toggle the following for keyboard access:
System Settings -> Privacy & Security -> Accessibility -> Terminal -> slide to allow
For Mac, this uses a custom keyboard library. This is built for a Mac keyboard, but you can also swap e.g. Windows key for Command simply enough.