Sketch a doodle of a drum or a saxophone to conjure a multi-instrumental composition. Look right into a webcam, speak, and watch your mouth go bouncing across the screen — the input for a series of charmingly clunky chain reactions.

That is what visitors to the MIT Lewis Music Library encounter after they interact with two latest digital installations, “Doodle Tunes” and “Sounds from the Mouth,” created by 2022-23 Center for Art and Technology (CAST) Visiting Artist Andreas Refsgaard in collaboration with Music Technology and Digital Media Librarian Caleb Hall. The residency was initiated by Avery Boddie, Lewis Music Library department head, who recognized Refsgaard’s flair for revealing the playfulness of emerging technologies. The intricacies of coding and machine learning can seem daunting to newcomers, but Refsgaard’s practice as a creative coder, interaction designer, and educator seeks to open the sector to all. Encompassing workshops, an artist talk, class visits, and an exhibition, the residency was infused together with his unique humorousness — a mixture of vigorous eccentricity and easygoing relatability.

Learning through laughter

Refsgaard, who is predicated in Copenhagen, is a real maverick of machine learning. “I’m fascinated about the ways we will express ourselves through code,” he explains. “I prefer to make unconventional connections between inputs and outputs, with the pc serving as a translator — a tool might let you play music together with your eyes, or it’d generate a love poem from a photograph of a burrito.” Refsgaard’s particular spin on innovation isn’t about directly solving problems or launching world-changing startups. As a substitute, he simply seeks to “poke at what could be done,” providing accessible open-source templates to prompt latest creative ideas and applications.

Programmed by Refsgaard and featuring a custom set of sounds created by Hall, “Doodle Tunes” and “Sounds from the Mouth” show how original compositions could be generated through a combination of spontaneous human gestures and algorithmically produced outputs. In “Doodle Tunes,” a machine learning algorithm is trained on a dataset of drawings of various instruments: a piano, drums, bass guitar, or saxophone. When the user sketches one among these images on a touchscreen, a sound is generated; the more instruments you add, the more complex the composition. “Sounds from the Mouth” works through facial tracking and self-capturing images. When the participant faces a webcam and opens their mouth, an autonomous snapshot is created which bounces off the notes of a piano. To try the projects for yourself, scroll to the top of this text.

Libraries, unlimited

Saxophone squeals and digital drum beats aren’t the one sounds issuing from the areas where the projects are installed. “My office is close by,” says Hall. “So once I suddenly hear laughter, I do know exactly what’s up.” This latest sonic dimension of the Lewis Music Library suits with the ethos of the environment as an entire — designed as a campus hub for audio experimentation, the library was never intended to be wholly silent. Refsgaard’s residency exemplifies a brand new emphasis on progressive programming spearheaded by Boddie, because the strategy of the library shifts toward a concentrate on digital collections and music technology.

“Along with serving as an area for quiet study and access to physical resources, we wish the library to be a spot where users congregate, collaborate, and explore together,” says Boddie. “This residency was very successful in that regard. Through the workshops, we were in a position to connect individuals from across the MIT community and their unique disciplines. We had people from the Sloan School of Management, from the Schwarzman College of Computing, from Music and Theater Arts, all working together, getting messy, creating tools that sometimes worked … and sometimes didn’t.”

Error and serendipity

The combination of error is a key quality of Refgaard’s work. Occasional glitches are a part of the artistry, and additionally they serve to softly undermine the hype around AI; an algorithm is simply pretty much as good as its dataset, and that set is inflected by human biases and oversights. During a public artist talk, “Machine Learning and the Arts,” audience members were initiated into Refsgaard’s offbeat artistic paradigm, presented with projects equivalent to Booksby.ai (an internet bookstore for AI-produced sci-fi novels), Is it FUNKY? (an attempt to tell apart between “fun” and “boring” images), and Eye Conductor (an interface to play music via eye movements and facial gestures). Glitches within the exhibit installations were frankly admitted (it’s true that “Doodle Tunes” occasionally mistakes a drawing of a saxophone for a squirrel), and Refsgaard encouraged audience members to suggest potential improvements.

This open-minded attitude set the tone of the workshops “Art, Algorithms and Artificial Intelligence” and “Machine Learning for Interaction Designers,” intended to be suitable for newcomers in addition to curious experts. Refsgaard’s visits to music technology classes explored the ways in which human creativity could possibly be amplified by machine learning, and learn how to navigate the sliding scale between artistic intention and unexpected outcomes. “As I see it, success is when participants engage with the fabric and provide you with latest ideas. Step one of learning is to grasp what’s being taught — the following is to use that understanding in ways in which the teacher couldn’t have foreseen.”

Uncertainty and opportunity

Refsgaard’s work exemplifies among the core values and questions central to the evolution of MIT Libraries — problems with digitization, computation, and open access. By selecting to make his lighthearted demos freely accessible, he renounces ownership of his ideas; a machine learning model might function a learning device for a student, and it’d equally be monetized by an organization. For Refsgaard, play is a way of engaging with the moral implications of emerging technologies, and Hall found himself grappling with these questions within the strategy of creating the sounds for the 2 installations. “If I wrote the sound samples, but another person arranged them as a composition, then who owns the music? Or does the AI own the music? It’s an incredibly interesting time to be working in music technology; we’re stepping into unknown territory.”

For Refsgaard, uncertainty is the key sauce of his algorithmic artistry. “I prefer to make things where I’m surprised by the top result,” he says. “I’m in search of that sweet spot between something familiar and something unexpected.” As he explains, an excessive amount of surprise simply amounts to noise, but there’s something joyful in the likelihood that a machine might mistake a saxophone for a squirrel. The duty of a creative coder is to repeatedly tune the connection between human and machine capabilities — to seek out and follow the music.

“Doodle Tunes” and “Sounds from the Mouth” are on display within the MIT Lewis Music Library (14E-109) until Dec. 20. Click the links to interact with the projects online.