ProtoSnaps and Processing

1. Serial Interfacing and Sensor Calibration Materials by Lindsay Craig and Ben Leduc-Mills ProtoSnaps and Processing

2. ProtoSnap and Processing Class Overview • What is Processing? Installation and Intro • Serial Comm from Arduino to Processing • Calibration of Sensors • Sensor Variables • Serial Comm from Processing to Arduino • Controlling and Using Physical Aspects of ProtoSnap Interface • Controlling Serial Comm and Handling Specific Events

3. Processing? • Processing is a free, open source, cross-platform programming language and environment for people who want to create images, animations, and interactions. • Created in 2001 by Casey Reas and Ben Fry at the MIT Media Lab. • Downloads, updates, reference, forums, etc. at: http://processing.org

4. Installation • For Linux: • Download the .tar.gz file to your home directory, then open a terminal window and type: • Tar xvfz processing-xxxx.tgz • (replace xxxx with the rest of the file’s name, which is the version number) • This will create a folder named processing-1.5 or something similar. Then change to that directory: • cd processing-xxxx • and run processing: • ./processing • For Mac: • Double-click the .dmg file and drag the Processing icon from inside this file to your applications folder, or any other location on your computer. Double click the Processing icon to start Processing. • For Windows: • Double-click the .zip file and drag the folder inside labeled Processing to a location on your hard drive. Double click the Processing icon to start Processing. • If you are stuck go to http://wiki.processing.org/index.php/Troubleshooting for help.

5. Anatomy of a sketch A sketch is a file or project you create in Processing. When you first open up a new sketch it will be completely blank.

7. Setup() This function runs once, at the very beginning of your sketch. You will use setup to set up certain aspects of your sketch, makes sense right? This is a good place to put your sensor calibration code. (More on sensor calibration later.) Most importantly for this class you will begin Serial communication in the setup function. The setup function without anything in it looks like this:

8. Draw() • This function is where everything happens in your sketch. The draw loop is the portion of code that keeps repeating while the Processing sketch is open. Any animation, interaction or changes to the images or variables in your sketch will need to be programmed inside of this loop. The draw loop looks like this:

9. How to Make Basic Graphics Basics here

10. Reference • One very convenient way to access Processing’s Help Reference is to highlight a function or a word used in your code, right click and select Find in Reference (which brings you to the processing.org reference page):

11. Hooking into Processing We can send values from the ProtoSnap to Processing through an Arduino, and use these values to control things in Processing First step: Open a new Arduino sketch Like Processing, we have a setup() function In that function, we need to open Serial communication: Serial.begin(9600);

12. Hooking into Processing We now have to set up all pins to the right pinMode, in our setup function as well as pulling any internal pull up resistors HIGH. We’ll get to the establishContact() function in a bit, but don’t forget to put it in now.

13. Hooking into Processing In our loop() function, we need to read from all the sensors and send the values out the serial port by calling Serial.print We used a delimiter to separate the sensor values so that the sensor values will be easier to distinguish when we get them into Processing.

14. Hooking into Processing Here are the rest of the loop() statements. Don’t forget the ln in the last Serial.println(digitalRead (7)); statement. There is more than this in the code on your ProtoSnap, we’ll get into that later. For now these are the basics you need to send Serial from Arduino to Processing.

15. Receiving Data in Processing • import processing.serial.*; • Serial usbPort; • int [ ] sensors = null; • boolean firstContact = false; • void setup() { • usbPort = new Serial (this,Serial.list( ) [0], 9600); • usbPort.bufferUntil (‘\n’); • } Now that we’re sending data from the ProtoSnap, we need a way to receive it in Processing. Luckily, there’s a library for that. First, import the serial library. We’ll also need a Serial object to define which serial port we’ll be using, as well as an integer array for our sensor data. We also have a boolean variable to keep track of whether we’ve heard from Arduino or not. In your setup function, you need to initialize your Serial object, passing it the parent object (don’t worry about this) which port you want to use, and the baud rate. Make sure you pick the same baud rate that you defined in the Arduino sketch. The bufferUntil function just stores our incoming data in a buffer until we’re ready to do something with it.

16. Receiving Data in Processing Our next step is to define a SerialEvent function – this function automatically gets called every time the character in our bufferUntil() statement is read from the serial port. We then read a chunk of data into a String, trim whitespace, and split it using our delimiter character (told you it would be useful!) into our sensors[ ] integer array. This puts each sensor value into its own addressable place in the array. There is a println that should be printing out the sensors values it sees – try running the sketch to make sure you’re getting values.

17. Receiving Data in Processing There’s a lot going on here, so don’t worry if it doesn’t make sense at first. Basically, we check for the Serial Communication from Arduino. Once it reads a carriage return it checks to make sure the data string is not empty. If it’s got something, we split the string up by our delimiter character into an array. This lets us put each sensor value into its own variable. If you don’t put the array values into their own variables you will get a null pointer exception when you try to use them in your code.

18. Waiting for Serial Communication Establishment One of the first things you will have to do is create a conditional that keeps your game from starting while Processing initiates Serial Communication. To do this use the firstContact variable as a condition for changing the screen state. Only when you have made contact with the Arduino will your game move to the next section of code.

19. Using Received Data in the Processing Sketch: The Button Next let’s keep Processing querying another conditional that waits for the user to press the button on the ProtoSnap. To do this we will need to check a variable (gameOn boolean variable) that tells us whether we have started playing the game yet. While that variable is false (or an integer that indicates we have not started playing the game) Processing will be stuck in a while loop that looks for a button press. When the button is pressed Processing takes a sensor calibration reading and changes the gameOn variable to true. This allows us to begin playing the game.

20. Using Received Data in the Processing Sketch: The Photoresistor In the first iteration of my game I simply mapped the light sensor value to the height of my screen. But my character never got to the top of the screen and it made my character jump all over the place due to the speed at which the light sensor values change. Before we fix this let’s look at how we use the light sensor to control the Y position of the character. Basically the closer our hand is to the light sensor the lower the value is. This means that we mapped the lowest possible value (0) to the bottom of the screen and the highest possible value (1023) to the top of the screen. But this assumes that we have enough light to actually get that value of 1023!

21. Using Received Data in the Processing Sketch: The Photoresistor So first I changed the map values so that the character was more in sync with the light values of the room in which I was programming. The room was a little dim so I used an upper light value of 150. So I wrote this: Instead of this:

22. Using Received Data in the Processing Sketch: The Photoresistor Or to fix the issue of sensor calibration we can get a value from the light sensor at the beginning of the game that we compare to the current light sensor variable. This way we can compensate for a brighter or darker environment. Remember this in the menu function? upLightVal is our calibration variable. And then we can use upLightVal in our mapping code in the draw loop. But it still jumps around a lot, huh?

23. Using Received Data in the Processing Sketch: The Photoresistor We can use the light sensor to move the character instead of using it to decide the position of the character. This is a subtle but important difference. This code says two things: 1. If the light sensor is between zero and fifteen (and the character is not at the bottom of the screen) move the character down by five pixels. 2. If the light sensor is higher than five below the initial sensor calibration value (and the character is not at the top of the screen) move the character up by five pixels.

24. Using Received Data in the Processing Sketch: The Photoresistor This works pretty well, but sometimes we get a bad initial calibration value and then the gamer is destined to hang out either at the top or the bottom of the screen. Since we only have two inputs to work with we need to figure out a way to use the other input (button) to trigger taking a new calibration reading. In order to allow normal usage of the button during game play and sensor recalibration we will look for three rapid pushes of the button, something that would rarely happen during normal play.

25. Using Received Data in the Processing Sketch: Programming the Game From this point out it’s a question of game design, programming and using the variables you are receiving from Arduino in the proper manner. With more inputs this gets more exciting, but two simple inputs (one Analog and one Digital) is a great way to learn this stuff. Any questions about the game code? Next up we will learn how to send communication the other way, from Processing to the Arduino.

26. Sending Data from Processing to Arduino: The Processing Code Good news! Since you have already imported the Processing Serial library sending Serial Communication from Processing to the Arduino is pretty simple. There are two pieces you need, very similar to the Arduino to Processing communication, but in reverse. To send a Serial message from Processing all you have to do is find the place in your Processing sketch where you want to send a message and type the following line: Serial.write (variable) ; Where variable is the value you want to send to your Arduino. The next 4 slides show where the game uses the Serial.write(); function.

27. Sending Data from Processing to Arduino: Low Health This portion of the code indicates that the character is low on life. The character ‘D’ is sent to the Arduino each time draw executes if the character’s life is below twenty five.

28. Sending Data from Processing to Arduino: Hit Test This is the hit test code. There is a boolean variable in it so it will only send Serial Communication or decrement the health meter when the character is first hit with an obstacle and not the entire time the character and the obstacle are touching.

29. Sending Data from Processing to Arduino: Shield On This portion of the code indicates that the user is pressing the ProtoSnap button and the shield has been activated. The variable shieldSwitch is used to limit the amount of Serial Data sent to the Arduino so there is only a signal when the shield is first activated.

30. Sending Data from Processing to Arduino: Shield Out This portion of the code indicates that the user is pressing the ProtoSnap button and has tried to activate the shield but there is no shield left to use. The character ‘N’ is sent every time, there is no limiting variable.

31. Receiving Data from Arduino to Processing: The Arduino Code Declaring a Buffer Variable and Assigning it a Value Receiving information is usually the most difficult aspect of Serial Communication. First we need to declare a Serial buffer variable to hold the Serial Communication, do this before your setup or loop functions. For my game I am only sending one char at a time so I declare a char variable and fill it with a zero like this. char Signal = ‘0’;

32. Receiving Data from Arduino to Processing: The Arduino Code Declaring a Buffer Variable and Assigning it a Value Then I use the following lines inside the loop function to fill that variable with the char that Arduino reads over the Serial line. if (Serial.available () > 0) { signal = Serial.read (); } If you want to send more than one value at a time you will need to create a char array instead of a signal char variable and use a for loop to check each value in the array. You will also need to make sure that Processing sends a full array each time it sends Serial Communication or have Arduino clear the array each loop cycle so you don’t wind up with outdated data in your Serial buffer. In this case you will need to change the if loop to a while loop because otherwise Arduino won’t catch the entirety of the communication.

33. Using the Serial Variable: The Arduino Code Now that we have data in our signal variable we can use it to control aspects of the ProtoSnap. You can do this in a bunch of different ways. We will cover two different ways to do this. One of them is a switch case and the other is an if statement.

34. Using the Serial Variable: The Arduino CodeSwitch Case Remember to put single quotes around each variable as you create each case statement.

35. Using the Serial Variable: The Arduino CodeIf Statement One cool thing about using if statements is that you can check if to see if the signal variable is NOT equal to a certain value.

36. Limiting Communication An important thing to note is that often you will only want to send Serial Communication at the beginning or end of an event. Such as with the hit test. You may want to have something happen on the Arduino when your character is first hit with a bad guy, but the hit test will continue to trigger Serial Communication after the first contact. You can limit this on either side of the communication. We already saw how you would limit that in Processing, here is how you might limit that on the Arduino side. Declare a boolean variable And assign the variable

37. Handling Different Types of Input from the Same Sensor Put 3 presses of button here, combination for hue in menu select

38. Other Ways to Handle Sensor Data Weighted variables here Running average

39. Other Ways to Handle Sensor Data Various ways to weight variables here as well Running average with recent weighted

40. Using Physical Aspects of the ProtoSnap Interface to Affect the Sensors The photoresistor is right next to the RGB LED. It is possible to use the LED to affect the photoresistor sensor depending on what is happening in the game. You may have noticed that when you use the shield the RGB LED lights up causing the character to move up a little. Useful if you want the character to move a little erratically when it has a low health. Make sure the red LED doesn’t Instead of just turning the LED on send it turn off if character has low health a random value, remember to put this code outside of your Serial.available(); if statement because we want the value to change each loop, not each time Arduino receives data.

41. Sensors Dave pointed out that back in the day people were able to use a single button to send morse code. There are many different ways to stretch the interface capabilities of your sensors. Think about creating handlers for different distinct sensor values and combinations. What kind of special moves or easter eggs will the user discover if they mess around with the sensors long enough?

42. Questions?

43. www.sparkfun.com 6175 Longbow Drive, Suite 200 Boulder, Colorado 80301