The Music Box is a project about music and abstract tangible interaction.
Technology has moved into a 2D space filled with screens. Sure, VR and AR are making an effort to make hybrid spaces between the physical and digital world, but I still feel like you depend on a screen of some sort, and it honestly makes the interaction feel a bit meaningless to me.
I am offering an exploration on music arrangement, inspired by old-school computing elements and based heavily on tangible interaction.
Being a classically trained pianist and a Mechatronics Engineer myself, I feel extremely wholesome by being able to explore this intersection, and I hope that this invention leads to something bigger. This is a project I intend to keep working on, and gradually build on top.
Context
I am constantly looking for different ways of composing music. Before attending NYU ITP, I created a web app that shuffles your MIDI-keyboard notes with five different algorithms, so you hear unexpected sounds for the things you usually play called SoundShuffle. It has served me countless times when I'm looking for a fresh chord progression or melody, and it taught me a lot about interaction; it needs an overhaul as you can probably tell.
Another experiment is this Synth Randomizer I created. This app connects to a MIDI-keyboard, mouse or computer keyboard and by pressing on a button, a new synthesizer is created for you. I added some controls over the randomization, but it is a tool intended to help people explore sound synthesis with as little input as possible.
Both if these experiments has been extremely important in my creative development, and I've thankfully received great feedback on both of them.
This loop of creative music tools for people is something that keeps on pulling me, but both of them rely on a screen. I am getting rid of that for The Music Box.
Concept
The Music Box is an old-school-like computer that users insert punch cards into, and when they do so, a pre-recorded sample is played. There are three different types of samples: Rhythmic, Harmonic and Melodic. By mixing the cards at different slots, different arrangements are played and composed.
The cards go in different slots and there are five different types of cards, each one containing a pre-recorded sample of music. The box has three rows and four columns, giving a total of 12 possible slots. Above each slot, there is LED that turns on when a card is inserted and shows which sound is about to be triggered.
The rhythm section sits on the bottom row, harmony on the middle one and melody on the top one. Each column represents a 8/4 measure, and when the playhead reaches any given column, the sounds corresponding to a punch that is inserted inside a slot plays.
The cards themselves resemble punch cards of the early days of computing.
As mentioned above, there are 5 different types of card. Type A, B, C, D and a Stop card.
Type A through D trigger a sample and the type Stop triggers a silent measure.
How it works
You might be wondering how the computer detects the cards. As you can see at the top of the cards themselves, there are different teeth patterns. The 3-bit pattern closes a switch on the receiving surface where the card is inserted, and a Teensy 3.6 boards polls through six different multiplexers to obtain the inserted 3-bit digit. On each of the holes (representing a 1) there is a tiny neodymium magnet covered in conductive tape that forces a switch.
The surface on which the magnets stick to is consisted of a 3-layer acrylic sandwich that contains other 3 magnets and a conductive tape surface.
A working prototype has been tested and all of the magnets are properly closing their respective switch.
Sound
The samples were composed and recorded by me. They have a somewhat simple structure to them, but the musical content is complex yet easily digestible. What I mean by this is that, the sounds are not boring or too happy. They are being designed to be engaging and inviting, without being inaccessible to children or musically inexperienced adults.
As I composed the music, I always kept in mind that the sound has to match this machine. What does an instrument that looks like this sounds like?
In early tests, I used piano sounds and classical drum sets, but the results were somewhat off-putting. When you insert a punchcard to a machine that looks like a weird retro computer, you're not expecting to hear a piano, that's for sure.
Scaling Up
When the electronics and coding, fabrication, music and interaction aspects of the prototype were solid enough, I scaled up the project to the final form.
For the electronics and coding part, that meant growing the circuit to hold all 12 slots (36 channels).
In the fabrication department, I spent a lot of time refining the final cards and the main module. Patterns in the cards were not easy enough to remember, so I added a color reference to them.
The 3-layer main module was also built-up to hold for all cards.
Finally for sound, I held long recording sessions to make sure that the music was exactly what I was looking for.
The final music is inspired by the works of very talented artists. Jordan Rakei, Anothony Ferraro (Astronauts, etc), James Blake and Nils Frahm are some of them. I gathered some of their songs into a Spotify playlist that you can see here.
The samples are written on a 100 BPM and a B-flat minor scale.
Final result
The Music Box works pretty great!
I showed it on ITP's Winter Show 2018 and people enjoyed using it quite a lot.
The feedback was consistent: people love the magnetic feedback you get when you insert a card and how the music starts playing automatically regardless of where you put a card in.
The music proved to be accurate, and the interaction a fun and engaging one.
Personally, I loved seeing how some users kept using it until they figured out the algorithm. There are a few playback rules that purposely make it harder to decipher but they allow for an easier interaction, and many stayed on it until they could understand it perfectly.
