Week 1: Experiment

Or an attempt at one. After my research this week, I decided to make a simple app to test the experience of texting words into sounds. Nothing fancy. I just wanted a sketch to emit a different sound for each word submitted and if time, integrate that into a simple chatroom using web sockets.

I quickly realized that I knew next to nothing about making and working with sound in the browser. I taught myself about synths, envelopes, frequencies, octaves, amplitudes, and more. Remembering that the music peeps on the floor like Tone.js, I found some starter sketches on The Code of Music to help me with that library—thank you, Luisa!

Though I very much enjoy technical challenges, working with musical terminology and coding specific to this area took longer than I expected.

Currently my sketch can:

  • Receive words through an input field, count those words, and print the quantity in the console. Eventually this amount will pass into a function to emit the same number of tones.

  • Until then, a mouse press triggers three tones at random frequencies, each of a different duration, in order, and do not loop.

  • …but only after the first press, and then I hear nothing. However, I can see evidence in console that the function is executing properly.

  • Best guess right now is that I need to use objects and that with a class I can create a new Audio Context upon every mouse press (or eventually when new words are submitted). I need to spend some time learning more about the Web Audio API and talk with music people.

Most important, here are the questions that this process raised for me:

  • Um, do I really want to work with material and methods with which I’ve had very little experience so far (natural language processing and sound)?

  • How will participants know that the sounds are from their texts? They need immediate feedback from their actions to care. I found a video interview with Werthein in which he describes the development Samba Surdo project. He noted that people with sight wanted to see where the sound was originating.

  • It’s one thing to generate random sounds from words, but if it’s always random, then there is no meaning in the sonic translation. What if for each new word the associated random generated sound was saved and a participant constructs a new audible language—an abstract sound lexicon—exploring and learning as they go?

  • Does each new participant have their own “animal” sounds? I suppose I can represent this by placing users in different frequency ranges for now.

  • What if participants build a new audible language together?

  • What if the language is already coded for them, and participants need to figure out their words’ sounds?

  • Right now, my program recognizes a “word” as surrounded by spaces. What’s to keep folks from typing in gibberish? What if they use emojis?

  • Speaking to the above research, are sounds somehow related to the meaning of words? If so, how do I make that clear and understandable to participants? And again, what about gibberish submissions?

  • In general, what’s the story arc of the experience?

  • How might participants hold conversations if they are unsure how others’ texts are translated? How might I offer expressive possibilities with sound to convey intent and/or meaning?

Next Steps
Finish this chatroom prototype! Visit Luisa in office hours for assistance and her advice on moving forward on a sound-related project (scheduled).

Week 1: Research on Word Analysis and Related Projects

I am curious about creating a communication platform that translates our words into audible animal calls for conversational play.

While I’m excited by this interest, I’m struggling to find an underlying reason or compelling argument other than it might be fun to see and hear what happens. So I started answering questions from last week and tried to make something to get closer to an answer.

What are some ways to analyze words?
One of my goals is to use material that is meaningful to people. For the moment, I’ve chosen to work with words and spent some time this week researching and brainstorming ways to analyze meaning from this form of input. Right now I’m imagining text input that is magically (ok, computationally) sonified.

So what’s to capture about a word or a phrase?

What it means?
How it sounds?
The speed at which it was typed?
Do I… Count letters, syllables, words, parts of speech*
Identify keywords, most common words, unique words
Determine level of subjectivity (fact <—> opinion)
Determine polarity (positive <—> negative)
Determine emotion (joy, surprise, anger, disgust, fear, sadness)
Compare words across a corpus of text (TF-IDF and word vectors)

*How do our word choices, specifically the use of function words (pronouns, position words, and auxiliary verbs) reflect “our personalities, emotional states, social connections and thinking styles?” (reference from Analyze Words, a Pennebaker project)

Siraj Raval’s Sentiment Analysis – Data Lit #1
Shiffman’s Word Counting & Text Analysis Series
Shiffman’s N-Grams & Markov Chains Series
James Pennebaker’s The Secret Life of Pronouns and Ted Talk
Allison Parrish’s Introduction to Natural Language Processing and Word Vectors

Is there any work out there like this? What sound-related, participatory and preferably public installations can I find?
Blobchat by Yotam Mann and Sarah Rothberg
“Blobchat is an experimental platform for communicating – sort of – through sound. Initially debuted as a quadraphonic installation, Blobchat 1.0 invites you to try out conversation as performance. You can chat at other blobs on a state-of-the-art blobchat-enabled computer. As you chat, your words are sonified, and get garbled together with all the other chats into a collective stream of nonsense. But is it nonsense? We use *machine learning* to make new sense of the garbled blob. A new sense! Out of the collective gesture of chatting!”

Works by Daily tous les jours*:
Hello Trees!
“The installation takes the form of an illuminated arcade that extends from the ground to the seam of the tree canopies above. Passers­by are invited to input messages at stations located at each end of the promenade, and watch and listen as their voices are translated into light and sound patterns that travel along the arches as they walk beneath. The original input is gradually transformed from language to musical melodies. When two or more messages meet, a special light and sound effect is triggered.”

I Heard There Was a Secret Chord
“…a participatory humming channel that reveals an invisible vibration uniting people around the world currently listening to Leonard Cohen’s “Hallelujah”. The project comprises a room and a website. The room contains a simple numerical display of current online listeners, each represented by a humming voice in the space. Underfoot, these sounds are transformed into low frequency vibrations as visitors start humming along.”

21 Balançoires
“21 Balançoires (21 Swings) is a musical installation from which certain melodies emerge only through cooperation between players, thus stimulating a sense of community and ownership of space…. When in motion, each swing triggers different notes, and when used all together, the swings create a musical composition in which certain melodies emerge only through cooperation. This collaborative exercise stimulates intuitive play and experimentation amongst people of all ages and backgrounds, whether they know each other or not, and leads participants and spectators to become aware of each other, and their environment.”

Mesa Musical Shadows
“Musical Shadows is an interactive pavement that reacts to the shadows of passersby by playing sounds of singing voices. As visitors together explore different soundscapes with their shadows, they become part of a collective sound and body performance. This new scenario for public space engages strangers to bump up against one another and share a moment of magic igniting in them a sense of what is possible together.”

Works by Rafael Lozano-Hemmer:
Amodal Suspension
“One objective of the piece was to make a public spectacle by using the private medium of text messaging, slowing down communication and introducing the possibility of interception.”

Less Than Three
“…an interactive installation of light strips that form a network between two intercoms. As a participant speaks into an intercom, his or her voice is translated into corresponding flashes of light and this light pattern is transmitted visually along one of the several possible pathways through the network. When it reaches the other side, the viewer’s phrase is once again released as sound.”

Voz Alta
“…the megaphone amplifies the voice, a 10kW searchlight automatically ‘beams’ the voice as a sequence of flashes: if the voice is silent the light is off and as it gets louder so does the light’s brightness.”

Voice Array
“As a participant speaks into an intercom, his or her voice is automatically translated into flashes of light and then the unique blinking pattern is stored as a loop in the first light of the array. Each new recording pushes all previous recordings one position down and gradually one can hear the cumulative sound of the 288 previous recordings. The voice that was pushed out of the array can then be heard by itself.” (Similar Voice Tunnel in NYC)

Samba Surdo by Lucas Werthein
“Samba Surdo is an interactive eight-channel sound installation that allows viewers to experience Brazilian samba. Visitors are invited to walk into the center of a room where eight speakers are positioned around them in the shape of a circle. Each speaker represents a different instrument played in a samba orchestra. As the visitor moves throughout the circle, his/her position is tracked by a set of sensors that determine which speaker should turn on. In this way, Samba Surdo gives the visitor the role held by the Master of Percussion in a samba orchestra; by moving and experiencing the installation, the visitor determines which instruments are played.”
(reminds me of my spatial sound puzzle from last fall!)

*Thanks to Yeseul for pointing me in the direction of this Montreal-based design studio!

And just for fun, there’s no shortage of these:
Human voice changers
Humans making animal sounds (also compelling because of the humans’ expressions)
Animals making human sounds

Next Steps
Scour the internets for repositories of recorded animal sounds and learn more about how animal behaviorists decode animal communication.

Visit Sarah in office hours to learn more about Blobchat (scheduled).

Week 0: Initial Research

Part 1: What do I like to do? 
Over winter break I reflected on my work to date and sought out resources and references to guide my thinking about a thesis project. I researched artists and exhibitions and read critical writings on interactive art and design for audience engagement.

I found frameworks for understanding interactivity and through which I analyzed my own work at ITP.  In particular, Brigid Costello’s Play Framework, offered thirteen different characteristics (listed in no particular order here): camaraderie, competition, captivation, exploration, subversion, simulation, sympathy, sensation, discovery, difficulty, creation, fantasy, and danger. Considering my own projects within this model, these themes emerged: exploration, discovery, creation, camaraderie, competition, captivation, subversion, and fantasy.

My readings (see below) referenced many artworks, and I dutifully followed up on as many interesting leads as possible. Here are some artists and projects that resonated strongly with me:

Jeppe Hein’s work often appears in traditional gallery spaces or in public spaces and almost always respond to the presence or absence of viewers. Hein often takes something familiar and modifies it in many unexpected ways and regularly incorporates mirrors and reflections in his work.

Trends in the work of Rafael Lozano-Hemmer over the past couple of decades include the representations and amplification of presence through projection and the interception, translation, and transmission of communications. Like Hein, his installations sometimes respond to the presence (following or running away) or the absence of viewers in a space, often endearing them with anthropomorphic qualities. Several pieces encourage participants to directly interact with one another.

Ernest Edmonds creates generative works that evolve over time from interaction with participants. Edmonds is interested in how viewers can establish long-term relationships with artwork.

In Philip Worthington’s Shadow Monsters, shadows of participants turn into fantastical monsters (with accompanying animal sounds) that morph in response to playful movements and gestures.

Tine Papendick’s Digital Puppetry similarly incorporates visitor’s bodies, but this time in full color and allows them to playfully mix and collage representations of themselves with others.

Seiko Mikami’s Desire of Codes and rAndom international & Chris O'Shea’s Audience, are both works in which sensory devices respond, often in unison, in anthropomorphic ways to the presence of visitors.

From this research and reflection, I realized that I really enjoy making work with collective play and shared experiences, often using conversation or engagement with compelling content as opportunities for making connections with others. And if it wasn’t obvious to me before, it is now: work is meaningful to people if it incorporates aspects of themselves, such as their gazes, their expressions, their reflections, their presence, their postures, their movements, their speech, their words, their heart beats, their brain waves, their touch, and their choices.

Part 2: What would I like to do?
The work I did over break helped me determine that I want to create a participatory, collective, and playful experience that uses meaningful material and is accessible for all ages and by our entire community at ITP. 

A footnote of Act/React: Interactive Installation Art led me to the Encyclopedia of Animal Behavior, and I unexpectedly found myself in the chapters concerning animal calls and communication networks. I started wondering what it might be like for humans to talk in the voices and sounds of other species.

There’s something fun and unexpected about imbuing familiar material with meaning from elsewhere, and so as a starting point for my project, I am curious about creating a communication platform that translates our words into audible animal calls for conversational play. 

Part 3: What are some next steps?
With this starting point in mind, I need to move into the next phase of conceptual development. Here are some paths to follow, although I’m sure to uncover more questions as I go:

Research and brainstorm ways that words can be analyzed and represented. What is sentiment analysis and do I want to use any of those techniques? 

Find and collect repositories of animal recordings. Which animals? What are they saying, how do we know, and does that matter?

Is there any work like this? What sound-related, participatory and preferably public installations can I find?

Do I want to create one large installation piece or several smaller ones? Will it live online, in physical space(s), or both?

Starting prototyping and learning from small experiments!

Bianchini, Samuel, et al., editors. Practicable: From Participation to Interaction in Contemporary Art. The MIT Press, 2016.

Breed, Robert Stanley, and Janice Moore. Encyclopedia of Animal Behavior. Elsevier Academic Press, 2010.

Brown, Kathryn. Interactive Contemporary Art - Participation in Practice. I. B. Tauris, 2016.

Candy, Linda, and Ernest A. Edmonds. Interacting: Art, Research and the Creative Practitioner. Libri Publishing, 2011.

Candy, Linda, and Sam Ferguson, editors. Interactive Experience in the Digital Age: Evaluating New Art Practice. Springer International Publishing, 2014.

Cornell, Lauren, and Ed Halter, editors. Mass Effect: Art and the Internet in the Twenty-First Century. The MIT Press, 2015.

Costello, Brigid M. Rhythm, Play and Interaction Design. Springer International Publishing, 2018.

Fifield, George, et al., editors. Act/React: Interactive Installation Art. Milwaukee Art Museum, 2008.

Karahalios, Karrie, and Judith Donath. “Telemurals: Linking Remote Spaces with Social Catalysts.” Proceedings of the 2004 Conference on Human Factors in Computing Systems  - CHI ’04, ACM Press, 2004, pp. 615–22.

Kwastek, Katja. Aesthetics of Interaction in Digital Art. The MIT Press, 2013

Pop, Susa, et al., editors. What Urban Media Art Can Do: Why When Where & How. Avedition, 2016.

Seevinck, Jennifer. Emergence in Interactive Art. Springer International Publishing, 2018.

The Walrus. "The Art of Play | Mark Kingwell | Walrus Talks." YouTube, https://www.youtube.com/watch?v=H4wfLyOgM8E. Accessed 12 Jan. 2019.

Zimna, Katarzyna. Time to Play: Action and Interaction in Contemporary Art. I. B. Tauris, 2014.