Empirical studies of collective intelligence

Highlights of this work include:

Meta-analysis of when combinations of humans and AI are useful

Surprisingly, we found that, on average, human-AI combinations performed worse than the best of humans alone or AI alone. But human-AI combinations performed significantly better for creation tasks than for decision tasks. 

An international AI negotiation competition

In this work, human participants designed prompts for AI negotiation agents, and the AI agents then negotiated with each other. We found that principles from human-human negotiation theory remained crucial, even in AI-AI contexts. For example, warmth, a traditionally human trait, was surprisingly effective across all performance metrics. However, AI agents also performed better when using specific technical strategies with each other, such as prompt injection and strategic concealment.  

Studies of how using generative AI affects groups

In these studies, we found that people tend to prefer to work with AI systems rather than other humans when two conditions hold: (1) the AI is perceived as more capable than humans and (2) Personalization is perceived as unnecessary.

Generative AI helps creativity when two conditions hold: (1) the AI provides useful cognitive resources for performing the task and (2) humans have high levels of metacognitive strategies—the ability to actively monitor and regulate their thinking to achieve goals and solve problems. 

Generative AI systems—trained on textual data that are inherently cultural—exhibit cultural tendencies when used in different languages. For example, AI models queried in Chinese (versus English) are more likely to exhibit a more interdependent (versus independent) social orientation, and a more holistic (versus analytic) cognitive style.

Studies of how internal group structure affects group performance

In these studies, we found that when groups estimate quantities (such as investment values and opinion poll results), it is possible to improve the accuracy of the groups’ collective estimates by rewiring who talks to whom. For example, group accuracy improved when group members whose opinions were in the middle of the distribution received information from those with polarized views at the extremes.

Contrary to some previous research, we found that the average skill level of group members was more predictive of future group performance than skill diversity, cognitive style diversity, and social perceptiveness combined.

Other empirical studies of group performance