Robotic Piano Playback
After several weeks of casual spare-time research and implementation, I’ve finally built a fully working piano playback robot. The usage is simple: someone plays or sings an arbitrary monophonic melody, and the robot, parked on a piano bench, will play it back.
The physical construction of the robot is very simple: it consists of a car with a crane-like arm mounted on it. The arm is used to push down and release a single piano key. On either end of the car, there are sensors which detect if the robot has come too close to the edges of the piano bench. The simplicity of the robot comes at the price of significant limitations, such as only being able to play back melodies on the white keys.
Software is the challenging part of the project. The Mindstorms sound sensor is too primitive to use for pitch analysis. Without hacking it, you can only extract the amplitude of the sound signal, not any frequency details. Instead of the NXT sound sensor, I use a macbook pro and it’s built-in microphone. A computer separate from the NXT is involved, so an additional set of communication problems arose.
Here’s a rough outline of happens to make the playback work, from capturing the melody line to playing the melody back.
- On the mac, using a Python AppleScript bridge, the QuickTime Player is invoked and starts capturing audio.
- Once the audio is captured into an AIFF file, a very useful pitch detection library called aubio processes the audio file and extracts raw frequency-to-time data, sampled at some tick rate. Compiling this library on OS X was quite a feat!
- Next, the raw data is processed by throwing out extraneous values and extracting a melody
- Once we have the melody line, we inject it into an NXC program, and compile it with the nbc compiler
- This program is then sent via bluetooth to the robot via nxtcom
- Using NXT_Python and Lightblue Glue, the robot is told to execute the program.
Please let me know if you have any questions or suggestions! Here is the source code.
4 comments:
Brilliant! However, changing and recompiling a program whenever you wish to run it with a new set of parameters is seldom a good way to do things. You could save doing these steps by having the NXC program read the note and timing information from a file. Then all you have to do for each tune is to assemble the note and timing information into a data file and send that to the NXT. This would probably also speed up the process.
Hi Noel, I’m glad you like it, and thanks for the feedback!
Your suggestion depends on there being file I/O functionality on the NXT brick. I didn’t see any documented API to that effect, though I admit, I mostly skimmed the docs! The bulk of the time seems to be spent initializing the Bluetooth connection…
Nice implementation! At NI Week ‘06 (I think it was), one of the developers of the NXT-G software had a wonderful little robot on display. He used a DSP board to evaluate ambient pitch, and transfered this information to the NXT via the Hitechn ic protoboard digital inputs. The result was “whistlebot” – you could sing, whistle, or play an instrument, and the robot would move in repose (middle A might mean “go forward”, while C would be “turn left”, etc.). a really lovely implementation with a minimal of hardware, that I’d love to see pushed further – like what you have done here!
Thanks. That’s an interesting idea — I never considered using a dedicated DSP board.