Holly Herndon Is Making Music With an AI Baby

Holly Herndon has spent the past few years nurturing her baby. But unlike the Berlin-based musician and composer, the child isn’t human. Instead, it’s an artificial intelligence that merges Herndon’s voice and that of her partner, the academic Mat Dryhurst, into a synthetic female entity by building neural networks around audio samples of their voices. Herndon and AI expert Jules LaPlace created the baby, aptly named “Spawn,” to provide backing vocals for her acclaimed 2019 album Proto alongside a choral ensemble. On “Frontier,” a track inspired by Appalachian Sacred Harp singing, Spawn enters after 90 seconds, eerily mimicking a human’s voice augmented by a machine. 

Some speculate Herndon’s experiment spells doom for artistic intent amid rising apprehension that AI will render human talents obsolete, but she seems unthreatened—emboldened, even—by the technology. “The best way forward is for artists to lean into developments with machine learning,” says Herndon, who recently earned her Ph.D from Stanford University’s Center for Computer Research in Music and Acoustics. After fellow musicians Grimes and Zola Jesus beefed on Twitter about whether or not AI will kill live music, Herndon sensibly interjected: “AI most likely won’t replace musicians outright. AI tools will cut costs to make generic music, and there is commercial incentive to progress this.”

Fast forward three years, and Herndon has debuted Holly+, a tool that allows fellow musicians to upload an audio file and compose using an AI-remixed version with Herndon’s processed voice. (She demonstrates its potential in a TED Talk with singer Pher.) Much like drum machines made basic drumming accessible to musicians but haven’t replaced drummers, Herndon envisions a future where “generating convincing spoken and sung voices will soon become standard practice for artists and other creatives, as presaged by the popularity of celebrity vocal deepfakes already found all over YouTube.” 

The tool presents newfangled possibilities for music composition but also raises difficult legal questions: Who owns the audio’s intellectual property, Herndon or the composer utilizing her vocal deepfake? To answer this, Herndon is leaning on crypto and established a decentralized autonomous organization (DAO), a group of blockchain users that collectively make decisions similar to a worker co-op. When artists compose using Holly+, they submit work to the Holly+ DAO, whose members mint the finest submissions into NFTs that go up for auction. Proceeds from each sale are split among the creator, DAO members, and Herndon herself, potentially solving the music industry’s ongoing issue of remixing without credit or attribution.

“Music has an especially bad track record when it comes to attribution,” Herndon told Art in America. “If you look at sampling, for example, many people have been sampled and not been paid. There are some legal restrictions on sampling, but there are fewer when it comes to AI, and I think it’s going to get really ugly really soon. Before working with AI, I didn’t think much about the sovereignty of an individual’s voice and what that means politically or aesthetically, but it’s a huge question. You can deepfake so much so easily now. We have some big questions on our hands.”

Herndon and Dryhurst are also launching Source+, a tool that allows artists to choose whether or not their work can be used as training data for AI systems like the image generator DALL-E. Herndon hopes her efforts will stoke open-mindedness among artists who may otherwise feel overwhelmed by the advent of AI. “Most AI processes we’re familiar with now are groundbreaking ways of aggregating the products of human intelligence,” she says, pushing back against the narrative about the rise of robot overlords. She says she’s more worried about “democratically unaccountable transnational companies training us all to understand culture like a robot or AI,” as opposed to viewing it as a tool that facilitates artistic growth. 

The power of AI and machine learning continues to reveal itself in exhilarating ways, but experts believe it won’t replace the creative ingenuity of artists outright. As University of Washington computer science professor Aaron Hertzmann writes, “we’re lucky to be alive at a time when artists can explore ever-more powerful tools.” Herndon’s experiments are proof positive.