Typing with a Piano [video](youtube.com) |
Typing with a Piano [video](youtube.com) |
I'll try to address the comments on here, so ask any questions you have!
My approach is a tad different though. My approach works by using the microphone. And I have the code to extract notes from a microphone via a FFT. Allowing for typing with say a guitar.
Extracting single notes works quite well. I haven't gotten around to test chords just yet, but I think I have the fundamentals there.
I'm a bit short on musical knowledge and instruments so so far I've only tested it with my voice (ugh), synthesized sounds and partially filled water bottles.
Also I have a suggestion for a graphical interface that would make the learning curve easier and making it language/alphabet independent.
Please reply to this comment if you are interested in integrating this.
RE: integration with what I did - I'm hoping to release the source code one day, then you'd most likely be free to use that code however you want (I'll probably MIT-license it). But as for my involvement, I'd like to be done with this project.
Slight tangent--I've often wondered if the principles behind audio keyloggers, where you record typing sounds and try to reconstruct what was typed, could be used to make a wireless computer keyboard that never needs batteries. Seems like you're attacking an inverted version of that problem.
Interesting stuff, thanks!
Kudos!
Which can be seen in action here: https://www.youtube.com/watch?v=HrUNrb3ihl0
And no, no April Fools here ;-)
Q: Can you play a real song so that we can see what it types out?
A: This isn't as satisfying as you may think since the entire typing range is only about two octaves, and most songs exceed that range, so nothing actually gets typed when you play those notes. Anyway, here's Wonderwall: Iuiuigaezobzgbxgebiuiuigaezobzgbegz
(that's actually the right hand of Für Elise)
Video of the stenotype machine at the Italian Senate (audio in Italian) https://youtu.be/e-z1ZAsQjio?t=18
Wikipedia page about the writing method (unfortunately only available in Italian, you may try Google Translate) https://it.wikipedia.org/wiki/Metodo_Michela
How to build a Michela Stenotype machine by modifying a MIDI keyboard: https://www.youtube.com/watch?v=q9Y9jtOB7G0
OpenSteno SW that supports various Stenotype machines including the Michela machine built using a MIDI keyboard as seen above http://www.openstenoproject.org/
https://en.wikipedia.org/wiki/File:Telegraph_Keyboard.jpg
https://en.wikipedia.org/wiki/File:Printing_Telegraph.jpg
See the article for some words on the topic: https://en.wikipedia.org/wiki/Printing_telegraph
> Hughes telegraph devices, which also had piano style keyboards, were very popular in France, where there were likely many more piano and harpsichord players than telegraphers.
But I'm not too sure how useful it is for efficiency even after practicing so much with it; there are a lot of false positives and negatives since keypress velocity doesn't matter for regular typing. With enough data, you could figure out a specific velocity threshold for each key so that you make fewer mistakes, and then maybe it'd be more helpful.
It can be made in a mechanical way, like a tristate button -- off, slighly-on, on.
I also suspect that the keyboard layout is more standardized between keyboards than the force threshold would be.
(And, it occurs to me that we're essentially chording with the shift key.)
Someone needs to make a velocity-sensitive mechanical qwerty keyboard :)
Edit: apparently "bigram" is the word I was looking for instead of "pair of letters".
To do that same process for musicality would be challenging since your average English sentence contains so many different letters. I would probably do something like assign each letter a set of sounds, then pick one of those sounds based on the rest of the sentence.
Here is the twitch channel where he broadcasts daily the game development: https://www.twitch.tv/adam13531
Here is the game he is developing ("An online strategy game with a focus on automation"): https://play.bot.land
3 pedals and 2 feet gives you 6 I think.
I briefly tried to find a youtube video of someone using it, but no joy-- I guess it's too boring to watch someone type. :)
https://www.youtube.com/watch?v=sjtZxjEH6cM
(if you've not seen that video, it's a little mind-bending)
(it's not really worth watching if you're satisfied with the YouTube video )
In the long run as I wrote I am aiming to support chords so that'll be necessery.
Would AMDF make the response time quicker?
For what purpose are you doing pitch detection?
In the end, I don't think there are many practical applications to something like this. I only had one real problem that could potentially be solved by this—it's when I'm working in a music program with my piano and limited physical desk space. The keyboard would be out of reach, so I could press an escape chord on the piano to enter Keyiano Mode, type whatever I needed, then reenter Piano Mode. The learning curve could be decreased by making on-screen overlays to show you what each key would do at any given time.
But yeah, that's solidly in daydream territory since I've got bigger fish to fry for now.
Seems like we're thinking very similar on this problem as my approach would be to have a constant onscreen HUD.
Initially my target was gamepads and other things viable for Virtual Reality. To try to demonstrate the universality of the method however I am working on a musical input method. Where musical keys can turn into keystrokes.
When you think about it you'll find that having a HUD onscreen gives a lot of room to implement clever algorithms that can speed up the process. Think huffman trees and other things that'll make the necessary keystrokes compressed.
What I have in the pipeline as of now is an FFT based algorithm that extracts the notes from the microphone, opening up for all instruments basically. The FFT part was surprisingly simple when dealing with one note at least, the tricky part might be chords and noisy environments...
What's so cool about far out ideas like this that you do for fun is that someone somewhere might find a usecase for it. Say for changing note cheats while playing a guitar or something similar.
As for making something like this harmonically pleasing, it's been touched on here in other threads, but I think the whole project would require a redesign. If you were still aiming for a one-to-one mapping of piano keys to letters, then I think you would end up sacrificing both learning speed and typing speed.
