Micro:bit Note Sequencer With Paper Playground

This Instructable will show you how to create synthesizer instrument using a micro:bit and a Paper Playground program. Paper Playground is an open-source system that enables you to design tangible, multimodal experiences using your computer's web browser, a webcam, and printed pieces of paper.

To get up and running with Paper Playground, follow the instructions here. This project is an intermediate companion to the Instructable Micro:bit and Paper Playground Integration for Enhanced Digital-Tangible Experiences. Random Servo Position With Micro:bit and Paper Playground is another related Instructable based on this project.

The Paper Playground project was developed by the Craft Tech Lab and PhET Interactive Simulations at the University of Colorado, Boulder, funded by National Science Foundation Award #2119303.

At the end of this Instructable, we will have created a tangible music player for the micro:bit—a paper-based interface that will let us create and play a sequence of notes (a melody). The video below demonstrates one such melody.

1. Paper Playground system setup on your computer running in a WebBluetooth-enabled browser (such as Google Chrome)
2. MakeCode
3. BBC micro:bit v2
4. USB cable for connecting the micro:bit to the computer
5. Webcam
6. Color printer to print the paper program and markers
7. Scissors

To get started, we first need to set up the Paper Playground system. It may seem complex, but we just need the basic setup to get it up and running. Follow the step-by-step instructions here and open http:\\localhost:3000 in your preferred browser. If everything goes well, you will see the screen below with links to different parts of the Paper Playground system. Open each of Camera, Canvas, Display, and Creator links in new tabs or windows. The Creator is where we load and program Paper Playground, and Camera is where we set up the webcam view to interact with the physical paper programs. The Display and Canvas are used when we have the additional projector setup, and we will mostly leave them untouched in this example (however, they still need to be opened in their own tabs).

We will first outline the instructions to get up and running with the example, and then we will circle back to see how the system works in the "Under The Hood" sections.

Let's begin with getting the micro:bit ready to go. We will program the micro:bit using a MakeCode program attached below. This code communicates with the Paper Playground over Bluetooth to receive commands and play notes on the micro:bit (discussed in detail in the Under the Hood section).

Open MakeCode at https://makecode.microbit.org/ and click on the Import button on the right edge of the page to upload the MakeCode playper-microbit-music-v1.hex file attached below. Once you import, you will see the block-based program shown in the figure.

Before we download the code to the micro:bit, we need to make sure that it can communicate openly over Bluetooth. Click on the gear icon on the top right and choose the Project Settings option. In the settings page that opens up, check that No Pairing Required: Anyone can connect via Bluetooth option is selected and click the save button.

Now, we can download the code onto the micro:bit using the Download button on the bottom left. Follow the instructions on the page to pair your micro:bit to your browser and load the code. Once the download is finished, you will see a checkmark on the micro:bit LED grid.

Now we can move to setting up the project in Paper Playground. Go to the Creator tab and create a new project in the default-space with a name of your choice (e.g. microbit-music-instrument). Now, use the Load Project button to upload the microbit-music-instrument.json project file attached below. That should open up the Paper Playground project with program #1657 as shown above. Loading a project file will erase any existing programs in your Creator space, so it is important to start a new project. Click on Send to Playground to launch the project in Paper Playground.

Switch to the Camera tab. You will see #1657 Micro:bit PlayMelody in the list of programs on the right.

Paper Playground enables us to interact tangibly with a computer and micro:bit. In this project, a paper grid and colored dots will act as our instrument to play melodies on the micro:bit. Download and print the print-template.pdf attached below on a color printer in a Letter (8.5"x11") or A4-sized printer. You can leave the first page with the grid as is, but cut out and separate the red dots form the second page as shown below. The red dots are similar to the markers around the grid, just larger (at least twice the size). You can also create your own red dots using a Sharpie!

Set up your webcam using tripod or mount such that it faces down on your table and can see the printed paper program.

Now, go to the Camera tab on your computer and in the Camera drop-down menu on the right, check Enable Camera to see the view from your webcam appear on the left. Adjust the camera and/or the paper program such that the entire sheet is visible in the webcam view and you see a blue rectangle overlay on the paper—this means that Paper Playground can detect the program appropriately.

If you do not see the blue rectangle on the paper, you might need to adjust the lighting in your room (turn on the lights if it's dark) or calibrate Paper Playground. Click on the Calibration drop-down menu on the right follow the instruction given for each color, R(ed), G(reen), B(lue), and D(ark). Once you complete the procedure, Paper Playground should detect the paper program well and you should see the blue rectangle overlay.

We are one step away from playing with the instrument! Go to the Display tab and click the blue Connect to BLE button on the right side. You will see a list of available devices in a pop-up, scroll to find the BBC micro:bit-named device and select Pair. You should then see a confirmation message appear in the console window at the bottom, and a Bluetooth icon briefly appear on the micro:bit.

Once connected to Bluetooth, your micro:bit is ready to play music! Place the colored dots in different cells in the grid and you will hear the micro:bit play notes in a loop. The grid on the paper is a melody-creator where you can define a sequence of 8 notes for the micro:bit to play by placing the dots on the grid. Each row is to specify a note in the middle-C octave, and each column defines the note in the 8-note melody. Note that this simple melody player can only play one note at a time, so each column can only have up to one dot. A blank column will sound like a 1-beat pause in the melody.

The video below show how you can interact with the paper interface to create note sequences on the micro:bit. As you change the position of the dots, the melody gets updated and keeps playing in a loop.

Paper Playground is conceptually based on the model-view-controller framework of software. Take a look at Mozilla's quick explanation here.

In our Paper Playground project, we have one program #1657 named Micro:bit PlayMelody. Each program in Paper Playground corresponds to one paper program, which, in this case, is the melody grid we printed in Step 4.

As we can see in the image above, the program has a number of components in the Model and Controller sections. The Model components are like program variables that store data about the current status of the program, e.g. activeMelody which stores the sequence of notes based on the red dots placed on the paper. The Model components are used and controlled by the Controller components that actually execute code in the Playground, e.g. playActiveMelody is a timed loop that iterates over the notes in activeMelody and chooses the current note using setMelodyNote. sendActiveMelodyNote, on the other hand, sends the active setMelodyNote to the micro:bit over Bluetooth to play on the built-in speaker. To view details about any of the components, click on the pencil icon (Edit icon) located right next to the component.

This paper program also has a few things running behind the scenes as Custom Code, which offers greater customizability. In this case, we use custom JavaScript code to track the placement of red dots on top of the paper program and update the activeMelody accordingly. Click on Custom Code button on the bottom to view and modify the custom JavaScript functions.

Paper Playground and the micro:bit communicate with each other using Bluetooth, once we connected them using the Connect to BLE button in Step 5. The sendActiveMelodyNote Controller component in the paper program is responsible for sending a text message of the form playNote->(F) to the micro:bit for it to play the a note with the frequency F (e.g. 440 Hz for Note A). In this way, Paper Playground sends each note of the melody sequence, one after the other.

Click on the Edit icon next to sendActiveMelodyNote to see how we configure the Bluetooth communication. The micro:bit UART Bluetooth service enables the computer to send text-based messages to the micro:bit. At the bottom, we see the JavaScript code doing so using the writeStringToCharacteristic() function.

Once the micro:bit receives the text message from Paper Playground over Bluetooth, it interprets the message to play the appropriate music note. The main part of the MakeCode program we uploaded to the micro:bit in Step 2 is shown below. The program uses some Array and Text functions to retrieve the Paper Playground command of the form playNote->(F) and then checks to make sure that it is looking at the playNote command and nothing else. It then extracts the note frequency F from the command and calls the function playNote defined at the bottom of the code. This function first sets up the internal speaker on the microbit, and then uses the play block to play the specified note for 1 beat. Since the original Bluetooth message was text, we need to use parse to number to convert the frequency to an number value before playing it with the tone block.

An in-depth discussion of Bluetooth with micro:bit is outside the scope of this Instructable, but please refer to the following resources to learn more.

1. Bluetooth in MakeCode
2. Bluetooth implementation on the micro:bit
3. (Advanced) Bluetooth Info
4. Bluetooth UART services

Here are other activities to extend this project:

1. Experiment with different music blocks and timings in MakeCode to play different kinds of sounds in the playNote function.
2. Change the scale of the music instrument by changing the assigned note frequencies in the Paper Playground custom code functions
3. Come up with different types of music instruments and explore paper interfaces other than the grid melody creator.