While the purpose of the study was not to prove that AI systems are capable of replacing humans in creative roles, it raises philosophical questions about the characteristics that are unique to humans, says Simone Grassini, an associate professor of psychology at the University of Bergen, Norway, who co-led the research.

“We’ve shown that in the past few years, technology has taken a very big leap forward when we talk about imitating human behavior,” he says. “These models are continuously evolving.” 

Proving that machines can perform well in tasks designed for measuring creativity in humans doesn’t demonstrate that they’re capable of anything approaching original thought, says Ryan Burnell, a senior research associate at the Alan Turing Institute, who was not involved with the research.

The chatbots that were tested are “black boxes,” meaning that we don’t know exactly what data they were trained on, or how they generate their responses, he says. “What’s very plausibly happening here is that a model wasn’t coming up with new creative ideas—it was just drawing on things it’s seen in its training data, which could include this exact Alternate Uses Task,” he explains. “In that case, we’re not measuring creativity. We’re measuring the model’s past knowledge of this kind of task.”

That doesn’t mean that it’s not still useful to compare how machines and humans approach certain problems, says Anna Ivanova, an MIT postdoctoral researcher studying language models, who did not work on the project. 

However, we should bear in mind that although chatbots are very good at completing specific requests, slight tweaks like rephrasing a prompt can be enough to stop them from performing as well, she says. Ivanova believes that these kinds of studies should prompt us to examine the link between the task we’re asking AI models to complete and the cognitive capacity we’re trying to measure. “We shouldn’t assume that people and models solve problems in the same way,” she says.