Networked Hardware- a Tool for Tangible Networked Interactions

What is this?

Networked Hardware is an open source tool equipped with sensors, actuators, and wifi enabled microcontroller. It uses a node-base interface where you can link inputs to output, make it easy to connect boards wirelessly and without programming. It was designed with artists, performers, teachers and students in mind.

Some of the questions it our heads as we work on these:

• How can we make the Internet more tangible?
• How can performers and audiences connect in meaningful ways?
• How can we make the process more accessible?
• How can our bodies gain back their presence in digital world?
• Can we experience connections in tangible ways regardless of the distance?

Why?

We are all used to websites, online games, phone apps being responsive to our actions, being interactive. We are familiar to tapping, swiping, pinching, typing, pressing buttons and expecting things to change on a screen. Networked interactivity is prevalent, especially in the online realm. However, it is typically confined to the digital world and rarely extends to the physical one, and our bodies, our beings, are very physical.

Making networked physical devices can be daunting to a newcomer. In part because of the technical challenges but in a greater part because the amount of knowledge require across multiple fields.

• Electronics (sensors, actuators, batteries, voltage, etc)
• Physical Computing (Arduino programming)
• Network Communication
• Server-side programming
• Website (client-side) programming

It’s a lot, it’s a lot in a lot of fields, but we hope it doesn’t have to.

For who is?

Students 14+

We developed the board as a tool for ourselves and quickly realize the potential for others, since then we have iterated multiple times, incorporating the feedback of other performers and students. In particular, we want to thank School for Poetic Computation, for inviting us to be part of the Experiments in Network Performance class for the 3rd time. CultureHub for inviting us to teach workshops with their students, finally we want to thank Brooklyn College’s Performance and Interactive Media Arts Program, for including the board as part of the lesson core materials.

For this project you will need multiple boards, at least one, but the more probably the more fun. I would recommend at least one for ever 3 students. The classes we have taught have had 1 per student, but it is possible to work in small teams.

Once you are done setting things up, you won’t need many of these tools, but for now you might need:

• Soldering iron- we will order 90% of the components pre-solder/assembled already, but the ESP32 needs the headers soldered.
• Lead-free Solder
• Screw driver set
• Cutting pliers
• Micro-usb cable (charging and uploading code)
• Computer for uploading code

At it's core, the powerful ESP32 microcontroller with built-in Wifi, Bluetooth, and a 18650 Battery Module. We selected this mcu because of its wifi capabilities, it's battery charging circuit, and battery holder included.

The board is packed with sensors to detect changes in light, temperature, sound and even orientation and accelerations. It also has a knob and a button for tangible interactions.

As outputs and actuators, the integrated servo motor, relay, and analog (PWM) output expand the possibilities for physical interactions.

For advance users, the board also offers serial communication to other MCUs, reprogrammable GPIO pins and I2C communication though the popular STEMMA QT connector, compatible with many Adafruit and Sparkfun modules. 

I’m starting by the end here, I realized that all the text above has been a little abstract and maybe this is a good place to show you what I’m talking about. I’ll show you more later, but at least some examples now to illustrate where we are heading.

Each board will be pre-programmed to connect to Node-Red, we will cover this in a following step, for now I want to talk about how to connect inputs without having to re-program the boards. This is what you and your students will be doing to design your interactions.

Node-Red is a node based programming interface, this means that instead of writing code and functions, you link them together visually. You can still write code if you need to, but from a basic level, you don’t have to.

We made up our own node to represent a board. As you can see, they have multiple inputs and multiple outputs. If you hover over the outputs, you can see what they represent.

[image 1]

Make sure to update the name of the board to your boards name, to do this you can double-click on the board, this will open the side menu where you can type the name. Click Done and then Deploy. Every time you make a change, remember to “Deploy” it for it to take effect. 

[image 2]

To make connections, simply click and drag from one node to another. In this example I connected the photo value of board_1 to the servo of board_2. Now covering board one will make board 2 move! cooooool.

[image 3]

Feel free to play around with different connections, you can also connect a board to itself, for example here we are connecting the microphone of the board to its own relay, so if it hears a load enough sound, it will trigger the relay.

[image 4]

One very useful node is the debug node. You can use it to print the readings to the debug panel on the right. You can use the debug node to verify things or to troubleshoot them.

[image 5]

*a note about values: because we want to be able to connect anything to anything, we need the values to make sense to each other. How would you map a light reading to a servo movement? To address this all values are “normalized”, meaning we have mapped them form 0 to 1. So no light would be a 0, tons of light would be a 1. If the servo gets a value of 0 it will move all the way left, if it gets a value of 1, it will move all the way right, anything in between? Well it will be somewhere in between. 

---

For more advance users, we can also connect the boards to other servers or websites using web sockets, this means that you can connect a p5.js sketch or a website to your board!

This is a bit more involved so I’ll skip it for now, but feel free to reach out if you need help setting it up.

The end! This would be the end if you were teaching your students how to set up the boards, in reality we need to do some of the ground work for them so they can play with them, the rest of the instructable will cover that.

Ok, this is honestly the hardest part, and although it really isn’t that hard, I’m still struggling to make it easier. If you have some ideas please reach out!

Here is a block diagram of the back-end.

We will setup a Node-Red Server, this will be the place where we can connect to the other boards and the rest of the interent.

The boards use the MQTT protocol to communicate, it is a fast way to send data without consuming tons of bandwidth, great for IOT devices or our boards. They will connect to a MQTT Broker (Mosquitto), this will then connect to the Node-Red server where we can do the fun things. For advanced users, we can also use this Node-Red server to connect to other servers or websites.

I wrote a step-by-step in depth guide here on how to do it as well as some more advanced use cases.

When we get the boards from the manufacturer, we need to upload the firmware once, to do this we will use the Arduino Cloud IDE, I found that it is the easiest way to share arduino code and import the libraries needed. If you haven’t used the Arduino Cloud before, you will need to also install the agent to be able to program the boards.

The Arduino code can be found here. The only thing you need to change is the name of each board, that way you can differentiate them later. Here you can also change the network (wifi) settings, that way you don’t have to set them later which would take a minute.

The Arduino code should be ready to be uploaded, you can think about it as the firmware for the boards, once uploaded you don’t really have to change it again, unless you would like to add your own sensor or features later. But that would be beyond the scope of this instructable. Because of this, I am not going to explain the code line-by-line, instead I’ll describe it broadly.

1. The board is turned on, it will attempt to connect to wifi
2. When successful, it will connect to the server and subscribe so it can send and receive data
3. Then it will go into a loop, it will check if there are any new messages from the servers, if there are, it will update the actuators accordingly.
4. If there aren’t any new messages from the server, it will read the sensors.
5. If the sensor readings change (beyond a threshold margin), it will publish the data back to the server
6. Regardless if the sensor readings are different or not, it will check if it is trying to save battery or not, if it is, it will go to sleep for 20 minutes. If it isn’t, it will go back to the loop.

Perhaps the most important part?

We designed the boards on KiCad, a free and Open Sourced software for PCB design. Here is the link to the files in case you want to download them.

And here is the link to the manufacturing files, if you don’t want to make changes to the board, you can use these to order them directly from a PCB house.

You will still need to buy:

• The ESP32
• The battery
• The Servo

Total cost depends of how many you order. But it should be about $35 per each, on materials.

As you can see, this boards have been evolving for quite some time, and we want to push them further until they are a little more mature so they can stand on their own. So we are looking for collaborators.

We want to improve:

• The board itself, maybe it would have the microcontroller installed on the pcb itself instead of needing to buy a separate MCU. This would make it cheaper, and also much less time consuming. As a teacher you probably that’s a double win.
• Node-RED, so far we like it, but it is difficult to set up, and it can only support few students working on it at the same time, multiple users can interfere with each other and this can be frustrating to say the least.
• Feedback, we would love to hear what other teachers have to say. Do this sensors and actuators make sense? Would your students benefit from having others?

In the meantime, here is a video of a performance where the boards were controlling music, visuals, light, and of course, each other.