Week 8: Color Sound Pen Iteration 3

Generally speaking, colour is a power which directly influences the soul. Colour is the keyboard, the eyes are the hammers, the soul is the piano with many strings. The artist is the hand which plays, touching one key or another, to cause vibrations in the soul.
— Wassily Kandinsky, Concerning the Spiritual in Art, 1911

Your midterm must include a project with strong concept, enclosure, presentation (5-7min as you will present in class), sketches (if you have them), schematic (in Fritzing), code on Github, and documentation as post on blog. Technically, you must include one tech from each section. SECTION I: P5.js visualization, Servo as Output, Serial Communication, Serial Handshake, and Tone as Output. SECTION II: Digital Input (at least 2), Analog Output (at least 2), DOM based input, Bluetooth, and Web Server.

Inspiration & Considerations
This is the third iteration on my Color Sound Pen to explore the translation of color into sounds and a continuation of last week's progress. My goal this week was to create a completely wireless device, which meant using a smaller microcontroller, determining a battery solution, and again, figuring out the sound output. And really, I hoped to play a Kandinsky painting. I'd heard over the years (most recently from Luke DuBois at the 2016 Eyeo Conference) that the painter associated colors with different instruments. Could I locate his color-sound code and play a reproduction of one of his paintings according to how he might have heard it? (Remember also my Kandinsky mention in this Week 2 post?)

Part 1: Configure Adafruit Feather nRF52 Bluefruit with Adafruit RGB Color Sensor
I found an all-in-one microcontroller from Adafruit, the Feather nRF52 Bluefruit LE, an Arduino-compatible device with a USB Serial converter, plus Bluetooth Low Energy (BLE) capability. I learned that I could not send data through the serial monitor as I did last week--this was BLE and for serial, I would need Bluetooth Classic. However, I could potentially pair this device with my computer as a keyboard and then send keypresses to any program that might receive them, such as a text input field, a blank document, OR a P5 sketch! Once I got everything up and running, I would just need to update my Arduino sketch to send a different keypress for each color range the sensor recognized.

Bluefruit nRF52 Feather Learning Guide
hid_keyboard.ino (for pairing Bluefruit as a keyboard)
BLEHidGeneric.cpp (for BLE keycode definitions)
colorview.ino (for generating color hex values for r, g, and b and output to LED)

Materials & Tools
Adafruit Feather nRF52 Bluefruit LE
Adafruit RGB Color Sensor with IR filter and White LED - TCS34725
Diffused RGB LED - Common Anode
1x 1K Ohm Resistor
2x 560 Ohm Resistors
2x 4.7K Ohm Resistors
Wires & Jumper Wires
USB Cable Standard A to Micro-B
Customizable Perfboard
Stacking Headers
5x Male Header Pins
Permanent Marker
Soldering Iron
Container from Foil Washi Tapes (from Michaels)
2x #2 56 x 3/8" Machine Screws (cut to size with Electrician's Pliers)
1x Lithium Ion Polymer Battery - 3.7v 350mAh
Classic Crayola Markers
Metal LED Holder 5mm
5/16" Drill Bit

This took some time. First, I installed the Bluefruit, which entailed adding the board to the Arduino IDE, downloading a driver for my laptop to see the board at the USB port, and finally, installing a python wrapper in order to compile and flash sketches to the board. This involved quite a bit of troubleshooting, and the Adafruit Learning Guide was essential!

I ran the hid_keyboard.ino sketch first, which enabled me to pair the Bluefruit as a keyboard to my iPhone. With the connection established, I typed letters from the Arduino IDE serial monitor on my laptop (which was connected to the Bluefruit) to a blank note on my phone--so cool! Special thanks to ITP Resident, Jason, who gave me this idea in the first place!!

With that functionality established, I connected the RGB Color Sensor and RGB LED to the Bluefruit via a breadboard and ran the ColorView sketch (from last week) to ensure the sensor could still pick up color values, light up the RGB LED accordingly, and print the color hex codes needed to calibrate for colors on any surface. For this to work, I installed to 4.7K Ohm pull-up resistors to SDA and SCL pins. (These are  built-in to the Arduino 101 and Uno models with which I previously used but not present in the Bluefruit device.) With this working, I combined the ColorView sketch with hid_keyboard, and prepared to move everything off of the breadboard so that I could squeeze it into my enclosure (pictures below).

Here is the code for the realized third iteration demonstrated in the video below:
Arduino Sketch
P5 Sketch

*reference only, unable to install this Bluefruit device in the Arduino Web Editor at the time

Part 2: Playing Kandinsky
Key to this experiment was locating Kandinsky's color-instrument connections, which I found in his 1911 publication, Concerning the Spiritual in Art:

Red: “Light warm red has a certain similarity to medium yellow, alike in texture and appeal, and gives a feeling of strength, vigour, determination, triumph. In music, it is a sound of trumpets, strong, harsh, and ringing.” (Also, the tuba, according to Moszynska.)

Orange: “Orange is like a man, convinced of his own powers. Its note is that of the angelus, or of an old violin.”

Yellow: “Keen lemon-yellow hurts the eye in time as a prolonged and shrill trumpet-note the ear…”

Green: “In music the absolute green is represented by the placid, middle notes of a violin.”

Light Blue: “In music a light blue is like a flute…”

Darker Blue: “…a darker blue a cello;  a still darker a thunderous double bass…”

Darkest blue: "…and the darkest blue of all-an organ.”

Violet: "In music it is an English horn, or the deep notes of wood instruments (e.g. a bassoon)."

Brown: "But there remains brown, unemotional, disinclined for movement. An intermixture of red is outwardly barely audible, but there rings out a powerful inner harmony. Skillful blending can produce an inner appeal of extraordinary, indescribable beauty. The vermilion now rings like a great trumpet, or thunders like a drum."

White: “has this harmony of silence, which works upon us negatively, like many pauses in music that break temporarily the melody.”

Gray: “A blend of black and white produces gray which, as has been said, is silent and motionless…”

Black: “A totally dead silence, on the other hand, a silence with no possibilities, has the inner harmony of black. In music it is represented by one of those profound and final pauses, after which any continuation of the melody seems the dawn of another world. Black is something burnt out, like the ashes of a funeral pyre, something motionless like a corpse. The silence of black is the silence of death. Outwardly black is the colour with least harmony of all, a kind of neutral background against which the minutest shades of other colours stand clearly forward. It differs from white in this also, for with white nearly every colour is in discord, or even mute altogether.”

DuBois, Luke R. Eyeo 2016 – R. Luke DuBoisMore Unanswered Questions, Eyeo 2016, 10 Aug. 2016, vimeo.com/178369021.
Kandinsky, Wassily. Concerning the Spiritual in Art. 1911
Moszynska, Anna. Abstract Art. Thames & Hudson, 1990.

(Red) Tuba - sustain - E#3 from Samulis
(Orange) Church Bell from waterproofbox
(Yellow) [Trumpet] Fanfare from tobyk
(Green) Violin - C4 from MTG
(Light Blue) Flute-A4 from xserra
(Darker Blue) 5_Gis_Ais_cello_A_string from basta.poland
(Darkest Blue) Organ - loud - F4 from Samulis
(Violet) Bassoon C-4 Tenuto Vibrato from Carlos_Vaquero
Poster Reproduction of Vasily Kandinsky's Blue Painting from The Guggenheim

With a color-sound key in place, I decided to code these colors: red, orange, yellow, green, light blue, darker blue, darkest blue, and violet. I downloaded recorded instrument notes from Freesound.org, edited them in the free Audacity audio editing software, and set out to locate a possible painting to play. There were many Kandinsky reproductions from which to choose at the Guggenheim store; I choose his Blue Painting for the many cleanly defined areas of solid color.

My code for Playing Kandinsky:
Arduino Sketch for Printing Color Hex Codes to Serial Monitor*
Arduino Sketch for Playing Kandinsky*
P5 Sketch 
(P5 Sketch practice with DOM buttons)

*reference only, unable to install this Bluefruit device in the Arduino Web Editor at the time

Biggest take aways this week? Ensure adequate documentation exists for your new devices and ask for help, especially from the knowledgable ITP residents and faculty. It's a must when you're stuck. And especially when working on devices outside of our standard Arduino Uno starter kits. I originally ordered a different all-in-one microcontroller from Adafruit only to discover that it was really intended for developers and was accompanied with scant documentation. The little I did find was written at a level I could not easily follow. Contrary to it's advertisement on Adafruit's site, it wasn't even clear that Bluetooth Classic had even been implemented, yet--yikes! I had expected to use this to communicate to P5 through via serial. Though Bluetooth LE options were available--though not well documented that I could find, I learned after the fact that BLE did not have serial port profiles. Again, a special shout out to Jason for lending me his Feather nRF52 and suggesting that I use BLE to my advantage and connect it as a peripheral. Also thanks to Mathura who helped me consider how to build out my electric components on perfboard. 

What next? An on-off switch would be nice, as would the use of stranded core wire over relatively loose-fitting solid core jumper wires. I'm still curious about embedding a MP3 shield and speaker into the unit such that I can create multiples of the pen. Right now I'm bound to using only at a time with my computer--either via serial or BLE.