Mark Klein, PhD
Principal Research Scientist, Massachusetts Institute of Technology

Welcome Research Teaching Engagement Curriculum Vita Personal Info

Mission My research mission to develop technology that helps large crowds of human and software agents work together to solve our most critical and difficult real-world challenges.

Approach My approach is multi-disciplinary, drawing from artificial intelligence, social computing, data science, operations research, complex systems science, economics, management science, human-computer interaction, amongst others.

Projects I am developing new concepts for how computing technology can empower the different stages of the collective intelligence lifecycle:

Collective Sensemaking
Understanding complex problems
Collective Innovation
Developing solutions for complex problems
Collective Decision Making
Agreeing which solutions to adopt

Coordinating solution enactment

Contributions I believe that my most significant research contributions to date include:
  • The Deliberatorium, a unique and powerful integration of the large-scale participation of social media with the systematic structure of argument theory. This project has elicited world-wide interest and is tracked for commercialization.
  • Complex Negotiation. While AI and computational economics researchers have devoted considerable resources to enabling simple negotiations (i.e. with one or a few independent issues such as price and delivery date), I have been a world leader in investigating how negotiation can work in more realistic settings where there are many diverse stakeholders and many interdependent issues.
Both projects have become the core of active global research efforts, powered by students and professors who have contributed their time and energy freely based on their belief in the project's potential.

The Deliberatorium

Current social media fail badly when we try to engage large crowds in deliberating about how to solve complex problems, typically generating huge volumes of highly redundant disorganized content of very mixed quality, making it prohibitively expensive to find the "good stuff", as well as difficult to measure and improve how well the crowd meets the customer's needs. This problem plagues a broad swath of institutions, including news media, business, government, and NGOs.

Deliberatorium is a web-based system that combines ideas from argumentation theory and social computing to address this critical challenge. Under development since 2007, it has been used by thousands of individuals in such institutions as Intel, the Federal Bureau of Land Management, and the Italian Democratic Party.

Introduction to MIT Deliberatorium (10")
Introduction to the Deliberatorium (20" research talk at the 2012 Yale Epistemic Democracy Workshop)

Selected Publications
Spada, P., Iandoli, L., Quinto, I., Calabretta, R., & Klein, M. (2017). Argumentation vs Ideation in online political debate: evidence from an experiment of collective deliberation. New Media & Society.
Klein, M., & Convertino, G. (2014). An Embarrassment of Riches: A Critical Review of Open Innovation Systems. Communications of the ACM, 57(11):40-42.
Mark Klein, Paolo Spada, Raffaele Calabretta (2012). Enabling Deliberations in a Political Party Using Large-Scale Argumentation: A Preliminary Report. 10th International Conference on the Design of Cooperative Systems.
Bernstein, A., M. Klein and T. Malone (2012). Programming the Global Brain. Communications of the ACM, 55(5).
Gurkan, A., L. Iandoli, M. Klein and G. Zollo (2010). Mediating debate through on-line large-scale argumentation: evidence from the field. Information Sciences, 180:3686-3702.
Klein, M. and L. Iandoli (2008). Supporting Collaborative Deliberation Using a Large-Scale Argumentation System: The MIT Collaboratorium. Directions and Implications of Advanced Computing; Conference on Online Deliberation. University of California, Berkeley.

Popular Press
How Science Can Help Us Disagree, Sloan Management Review, Nature, New York Times, MIT Technology Insider (page 11), Information Week, MIT Tech Talk, The Independent (UK), Human Technology Journal, Next Generation Democracy (Chapter 4), Business Innovation in the Cloud (Chapter 9), finalist for 2011 Management Innovation Challenge

Intel, National Science Foundation, European Union FP7 program, John Templeton Foundation

Nicholas Adams, Jared Bataillon, Abraham Bernstein, Nancy Bordier, Marco Cioffi, Jeffrey Conklin, Gregorio Convertino, Anna de Liddo, Cristina Garcia, Ralf Groetker, Anatoliy Gruzd, Ali Gurkan, Luca Iandoli, Catholijn Jonker, James Lannigan, Chencan Xu & Yawen Li, Yiftach Nagar, Corey Nunez, Ivana Quinto, Hajo Reijers, Carlo Savoretti, Paolo Spada, Simon Buckingham Shum, Alex Smallwood, Catherine Spence, Mark Tovey, Cyril Velikanov, Michael Winikoff

Complex Negotiation

Current negotiation mechanisms fare well for the simple contexts they were designed for (e.g. a small number of parties, a small number of independent issues such as price) but are ill-suited to complex negotiations which include many parties, as well as many interdependent issues.

We are defining novel algorithms that help agents negotiate complex contracts with many interdependent issues, integrating ideas from nonlinear optimization as well as game theory. We have, further, been using machine learning techniques to determined which algorithms work best in which kinds of negotiation scenarios, by running simulated negotiations with thousands of systematicaly-generated negotiation scenarios.

An Introduction to Nonlinear Negotiation (9")

Selected Publications
Hoz, E. D. L., Marsa-Maestre, I., Gimenez-Guzman, J. M.,David Orden, Klein, M. (2017). Multi-agent nonlinear negotiation for Wi-Fi channel assignment. 16th International Conference on Autonomous Agents and Multiagent Systems
Aydogan, R., Marsa-Maestre, I., Klein, M., & Jonker, C. M. (2015). A Machine Learning Approach for Mechanism Selection in Complex Negotiations. Eighth International Workshop on Agent-based Complex Automated Negotiations (ACAN). Springer.
Marsa-Maestre, I., Klein, M., Jonker, C. M., Lopez-Carmona, M. A., & Aydogan, R. (2014). From Problems to Protocols: Towards a Negotiation Handbook. Decision Support Systems, 60:39-54.
Katsuhide Fujita, Takayuki Ito, Mark Klein (2014). Efficient Issue-Grouping Approach for Multiple Interdependent Issues Negotiation between Exaggerator Agents. Decision Support Systems. 60:10-17.
Zhang, S., Klein, M., & Marsa-Maestre, I. (2014). Scalable Complex Contract Negotiation With Structured Search and Agenda Management. Proceedings of the Twenty-Eighth AAAI Conference on Artificial Intelligence (AAAI-14).
Katsuhide Fujita, Takayuki Ito, Mark Klein (2012). A Secure and Fair Protocol that Addresses Weaknesses of the Nash Bargaining Solution in Nonlinear Negotiation. Group Decision and Negotiation Journal. 21(1):29-47
Hattori, H., M. Klein, and T. Ito (2007). Using Iterative Narrowing to Enable Multi-Party Negotiations with Multiple Interdependent Issues. Sixth International Joint Conference on Autonomous Agents and Multi-Agent Systems (AAMAS).
Ito, T., M. Klein, and H. Hattori (2007). Multi-issue Negotiation Protocol for Agents: Exploring Nonlinear Utility Spaces. Twentieth International Joint Conference on Artificial Intelligence (IJCAI).
Klein, M., P. Faratin, H. Sayama, and Y. Bar-Yam (2003). Protocols for Negotiating Complex Contracts. IEEE Intelligent Systems. 18(6):32-38.

National Science Foundation

Reyhan Aydogan, Yaneer Bar-Yam, Miguel Angel Lopez Carmona, Peyman Faratin, Katsuhide Fujita, Hiromitsu Hattori, Takayuki Ito, Catholijn Jonker, Ivan Marsa Maestre, Hiroki Sayama, Shelley Zhang

Idea Filtering with the Bag of Lemons

Open innovation platforms (web sites where crowds post ideas in a shared space) enable us to elicit huge volumes of potentially valuable solutions for problems we care about, but identifying the best ideas in these collections can be prohibitively expensive and time-consuming.

Our approach, called the "bag of lemons", enables crowd to filter ideas with accuracy superior to conventional (Likert scale) rating approaches, but in only a fraction of the time. The key insight behind this approach is that crowds are much better at eliminating bad ideas than at identifying good ones.

Selected Publications

Cris Garcia

Artifact-Centric Knowledge Sharing

How can we increase the efficiency and completeness of knowledge-sharing around highly complex technical artifacts such as airplanes and oil refineries? The sheer scale of the knowledge to share overwhelms conventional (keyword-based) search schemes, resulting in low retrieval precision and accuracy, and thus poor knowledge sharing, missed opportunities for improvement, and even unaddressed safety problems.

This project explores how the knowledge about a complex artifact can be indexed as semi-formal models attached to digital descriptions of the artifact itself. Documents with knowledge about a turbine blade can, for example, be hotlinked to the part of the CAD model that represents that blade. Or they can be hotlinked to elements in the process model which describes how the turbine works. This project has developed semi-formal rationale capture languages, intuitive user interfaces to enter the knowledge, and search tools to help people find relevant knowledge.

Selected Publications
Mark Klein (1997). Capturing Geometry Rationale for Collaborative Design. Sixth International Workshop on Enabling Technologies: Infrastructure for Collaborative Enterprises (WET ICE).
Mark Klein (1993). Capturing Design Rationale in Concurrent Engineering Teams. IEEE Computer. (26)1:39-47.

Siemens, British Petroleum, US Army Research Lab, Boeing

Prof. Abraham Bernstein

Conflict Management in Collaborative Design

Complex artifacts, ranging from physical artifacts like airplanes to virtual artifacts like software or process models, are created by many, sometimes thousands, of designers working on different but interdependent parts or concerns. If conflicts between these activities are detected late, or resolved imperfectly, this can have a huge negative impact on the timeliness, quality, and cost of the finished product. Existing conflict management approaches (e.g. change memos, multi-functional product review meetings, design-build teams) are however slow, expensive, and error-prone.

This project explores how AI technologies such as constraint management and diagnosis can greatly increase the speed and effectiveness of conflict detection and resolution in large-scale collaborative design settings.

Selected Publications
Mark Klein (2000). Towards a Systematic Repository of Knowledge About Managing Collaborative Design Conflicts. International Conference on AI in Design.
Mark Klein (1991). Supporting Conflict Resolution in Cooperative Design Systems. IEEE Transactions on Systems, Man and Cybernetics 21(6).
Mark Klein (1989). Conflict Resolution in Cooperative Design. The International Journal For Artificial Intelligence in Engineering. 4(4):168-180.

National Science Foundation, Boeing

Designing Robust Systems

Complex systems are typically designed by describing the normative flow of events in an ideal world, and later augmenting it in an ad hoc way to include steps for handling (anticipating and avoiding, or detecting and resolving) any exceptions that may occur. This unsystematic process can be seriously deficient in terms of addressing all important possible exceptions, and in using best practices to deal with them. Such errors in process design have brought companies to bankruptcy (cf Barings Bank).

This project has been developing knowledge-based tools that help designers analyze normative process models, systematically enumerate possible exceptions, and suggest best practice techniques for handling these exceptions. The approach is based on a growing taxonomy base of widely-usable best-practice strategies for detecting, diagnosing and resolving exceptions.

Selected Publications
Mark Klein and Chrysanthos Dellarocas (2000). A Knowledge-Based Approach to Handling Exceptions in Workflow Systems. Computer-Supported Collaborative Work. 9(3):399-412.

NSF, Defense Logistics Agency, Neptune Technologies, British Telecom, Hewlett-Packard, Boeing, University of Lecce (Italy)

Chrysanthos Dellarocas

Emergent Dysfunctions in Socio-Technical Systems

Large-scale socio-technical systems, made up of many interacting human and/or machine components, now operate at unprecedented levels of scale, speed and interdependency, and thus can be prone to highly dysfunctional emergent dysfunctions (e.g. failure cascades, thrashing) that are new to our experience and thus difficult to anticipate and avoid, or detect and resolve.

This project has been cataloging the ways complex socio-technical systems (e.g. for resource sharing or collaborative decision making) can get mired in emergent dysfunctions, as well as developing novel techniques for avoiding or resolving such problems.

Selected Publications
Klein, M., R. Metzler, and Y. Bar-Yam (2005). Handling Emergent Resource Use Oscillations. IEEE Transactions on Systems Man and Cybernetics A 35(3):327-336
Mark Klein, Juan-Antonio Rodriguez-Aguilar and Chrysanthos Dellarocas (2003). Using Domain-Independent Exception Handling Services to Enable Robust Open Multi-Agent Systems: The Case of Agent Death. Autonomous Agents and Multi-Agent Systems. 7(1):179-189.


Yaneer Bar-Yam, Chrysanthos Dellarocas, Juan Antonio Rodriguez-Aguilar, Hiroki Sayama

Deliberation Analytics

How can we monitor and guide open crowds, when they are deliberating in peer-to-peer contexts, so that their aggregate efforts progress efficiently towards good, well-evaluated solutions?

We are developing powerful analytics that datamine the digital traces of crowdsourced deliberation engagements to assess how well the deliberation is going, understand the skills and styles of the crowd members, and guide the crowd's attention to better achieve the goals of the deliberation customer. This project builds upon a semi-automated, crowdsourced approach to tagging the deliberative content of social media interactions, and is using the resulting coding to derive powerful new types of social network analysis (e.g. to assess the degree of balkanization, or bias, in a community).

Deliberation Metrics

Selected Publications
Klein, M. (2012) Enabling Large-Scale Deliberation Using Attention-Mediation Metrics. Computer-Supported Collaborative Work, 21(4-5):449-473.
Klein, Gruzd & Lannigan (2017). Using Deliberation-Centric Social Network Analysis to Assess Interactive Signed Ties. Proceedings of Sunbelt. Beijing, China.
Klein, M. (2015). The Catalyst Deliberation Analytics Server. MIT Technical Report.

EU FP7 program

Anna de Liddo, Anatoliy Gruzd, Luca Iandoli, James Lannigan, Hajo Reijers, Simon Buckingham Shum

Pareto-Centric Collective Decision-Making

This project is exploring how we can integrate advanced techniques for collective innovation, idea filtering, and complex negotiation to engage large crowds in finding
pareto-front solutions to complex and contentious problems. This direction promises high potential impact in such domains as participatory democracy and collaborative design. At a technical level, this work will require innovations across the collective problem-solving lifecycle, including crowdsourced moderation, crowd analytics, adaptive incentive mechanisms, idea filtering, complex negotiation, and algorithmic report generation.

Selected Publications
Klein, M. (2017). Towards Crowd-Scale Deliberation. Working Paper, MIT Center for Collective Intelligence.


One of the biggest challenges facing 21st century institutions is figuring out how to mine useful insights from the vast data sets now available to us. Current data mining technologies allow us to generate compressed views of large data sets, but lack people's ability to figure out what they mean and thereby create actionable intelligence. My goal is to achieve this by engaging crowds in scouring big data for insights, creating, building upon and critiquing each other's hypotheses about what the data has to say. This capability can then be used for applications ranging from business and government data analytics to intelligence analysis.

Deliberative Prediction Markets

Prediction markets represent a promising technique for assessing the likelihood of future scenarios, but are limited by the fact that crowd members do not share the reasons and evidence underlying their judgments, so participants can often make poorly-informed assessments. The goal of this project is to develop a deliberative prediction market framework wherein crowd members, supported by data-intensive analytics, share and evaluate the rationale for their assessments, in addition to just entering scenario probability estimates.