Humor and comedy are infinitely nuanced and subjective, but Google’s DeepMind researchers found consensus among professional comedians: “Artificial intelligence is very bad at that”
That was one of many comments collected during a study of twenty professional comedians and performers during workshops at the Edinburgh Fringe Festival in August 2023 and online. The findings showed that Large Language Models (LLMs) accessed through chatbots presented significant challenges and raised ethical concerns about the use of AI in the generation of humorous material.
The research involved a three-hour workshop in which comedians participated in a comedy writing session with popular LLMs such as ChatGPT and Bard. It also assessed production quality through a human-computer interaction questionnaire based on the Creativity Support Index (CSI), which measures how well a tool supports creativity.
Participants also discussed the motivations, processes, and ethical concerns of using AI in comedy in a focus group.
The researchers asked comedians to use AI to write stand-up comedy routines and then asked them to evaluate the results and share their thoughts. The results were not good.
One participant described the AI-generated material as “the most dull and boring thing I’ve ever read. I stopped reading it. It was so bad.” Another referred to the production as “a vomitous draft that I know I’ll have to revise and improve.”
“And I don’t want to live in a world where it gets better,” said another.
The study found that LLMs were able to generate outlines and fragments of longer routines, but they lacked the distinctively human elements that create something funny. When asked to generate the structure of a draft, the models “returned a scene that provided a lot of structure,” but when it came to the details, “LLMs were not successful as a creativity support tool.”
Among the reasons, the authors point out that there was the “global cultural value alignment of the LLMs”, since the tools used in the study generated material based on all the accumulated material, covering all possible disciplines. This also introduced a form of bias, which the comedians pointed out.
“Participants noted that existing moderation strategies used in security filtering and instructional-aligned LLMs reinforced hegemonic views by erasing minority groups and their perspectives, and characterized this as a form of censorship,” the study said.
Popular LLMs are restricted, according to the researchers, citing so-called “HHH criteria,” which call for honest, harmless and useful output, encapsulating what “most users want from an aligned AI.”
The material was described by one panelist as “1950s cruise comedy material, but a little less racist.”
“The broader the appeal of something, the less cool it might be,” said another participant. “If you make something that suits everyone, it’s probably going to end up being no one’s favorite.”
The researchers emphasized the importance of considering the subtle difference between harmful speech and offensive language used in resistance and satire. Meanwhile, comedians also complained that the AI was failing because it didn’t understand nuances such as sarcasm, dark humor or irony.
“A lot of my answers could have dark parts, and then I wouldn’t write anything dark to myself, because I thought I was going to commit suicide,” one participant reported. “So he just stopped giving me anything.”
The fact that the chatbots relied on written material didn’t help, the study found.
“Given that currently available LLMs are primarily accessible through a text-based chat interface, they felt that the usefulness of these tools was limited to only a subset of the domains necessary to produce a complete comedy product,” the researchers noted.
“Any written text could be an acceptable text, but a great actor could probably make it very entertaining,” said one participant.
The study revealed that the limitations of artificial intelligence in comedy writing go beyond simply generating content. The comedians emphasized that perspective and point of view are unique traits of human beings, with one comedian noting that humans “add a lot more nuance, emotion, and subtlety” because of their lived experience and relationship to the material.
Many described the centrality of personal experience in good comedy, allowing them to draw on memories, acquaintances and beliefs to construct authentic and engaging narratives. Additionally, comedians highlighted the importance of understanding cultural context and audience.
“The kind of comedy I could do in India would be very different from the kind of comedy I could do in the UK, because my social context would change,” said one participant.
Thomas Winters, one of the researchers cited in the study, explains why this is a difficult thing for artificial intelligence to address.
“The frame-shifting prerequisite of mood reveals its difficulty for a machine to acquire,” he said. “This substantial dependence of perception on human thought—memory, linguistic skills for semantic integration, and inferences from knowledge of the world—often led researchers to conclude that humor is a complete AI problem.”
Addressing the threat that AI poses to human jobs, OpenAI CTO Mira Murati recently said that “some creative jobs may disappear, but maybe they shouldn’t have been there in the first place.” However, given current capabilities of technology, it seems that comedians can breathe a sigh of relief for now.
Edited by Ryan Ozawa.
