Roberto’s Web Page

Aggregate Confusion

Aggregate Confusion

Aggregate Confusion is  a research project within the MIT Sloan Sustainability Initiative. Our aim is to help improve the measurement of firm’s ethical behavior. In particular, we aim to enhance measurements about environmental impact, labor practices, social impact, product safety, and governance.

For more on the MIT Sloan Sustainability Initiative click on the logo

What is the Aggregate Confusion Project?

The challenge of evaluating the environmental and social impact of companies’ operations and products has become a first order concern in the world. For instance, consumers would like to purchase products from companies that produce in ways that respect certain values – such as environmental protection, women empowerment, absence of child labor, etc.; governments would like to understand the impact companies have on communities – either for regulation or guidance; and investors would like to know if they are financing activities that might pose a reputational risk.

As of today, agencies assign environmental social and governance (ESG) ratings to firms in order to gauge to what extend firms behave in a socially responsible way. But these ratings diverge substantially between different rating agencies.  In fact, if we take two of the top five ESG rating agencies and compute the rank correlation across firms in a particular year, we are likely to obtain a correlation of the order of 10 to 15 percent. At least the correlation is positive! It is very likely (about 5 to 10 percent of the firms) that the firm that is in the top 5 percent for one rating agency belongs to the bottom 20 percent for the other. This extraordinary discrepancy is making the evaluation of social and environmental impact impossible.

In this regard, how can the world eliminate discrimination in the labor force if we are unable to measure it? How can industries produce safer goods and services for consumers if we can’t agree on how to size their impact? If we want managers, consumers, governments and investors to take the right decisions, we need to provide them with a unified guidance. This is the purpose of Aggregate Confusion.

Technically speaking, we will disentangle the three sources of divergence of ESG ratings: different aggregation procedures, different information sets, and different approximations for the underlying aspects they are interested in measuring. In other words, we want to quantify how much could be improved by taking a different approach to the already existing data. And with that we hope to reinvigorate the debate on how to improve those ratings.

Disentangling Discrepancies

In line with prior studies, we have corroborated the very low correlation of firms across different ESG ratings. Correlations as low as 10% in some cases. The disagreement is so extreme that a top performer in one rating can be rated as a laggard by another rating agency. In order to tackle this problem, we aim to answer what are the sources of discrepancies: how much is due to different methodologies and how much is due to different proxies.

Measurement Challenge

How can we improve measurement for ESG? We believe that a significant source of the discrepancy across rating agencies is due to measurement problems. In fact, most rating agencies would like to measure similar attributes and characteristics of firms. For example, how likely is the company to discriminate? To measure such attribute rating agencies use proxies. We are working in several areas to improve the current measurements. As we uncover new procedures to do so, we will make the data gathering and methodologies open source. Stay tuned!

Areas we are working on:

  • 1. Discrimination and Women Empowerment
  • 2. Water Management
  • 3. Food Waste
  • 4. Corruption and political divisiveness
  • 5. Household Stress
  • 6. Hidden Networks
  • 7. Job Quality

Principles behind the new measures

  • Open Source:                                       Open Measurement and Appeal Process
  • Non-Intrusive:                                    Data Collection does not rely on Standard Surveys
  • Continuous Process:                        Emphasize Process and Timeliness of Index
  • Privacy Protection:                           Individual’s differential privacy loss is protected
  • Imperfect Measurement:            To satisfy the previous four, the measurement needs to be nois


  • 1. Aggregate Confusion: The Divergence of ESG Ratings.


email the team:



Twitter @

Aggregate Confusion Team

Aggregate Confusion

Florian Berg

Postdoctorate MIT

Research Interest: ESG Measurement Discrepancies

Jullian Koebel

Co-Founder of the Aggregate Confusion Project, MIT

Research Interest: Water Management, ESG Measurement Discrepancies

Roberto Rigobon

Society of Sloan Fellows Professor of Management

Co-Founder of the Aggregate Confusion Project, MIT

Research Interest: Economic, Social and Ethical Measurement, International Economics, Monetary Economics

MIT Sloan Sustainability Initiative Team

Dominic Farello

Program Assistant for the Sustainability Initiative

Jason Jay

Director of the Sustainability Initiative

Bethany Patten

Associate Director of the Sustainability Initiative.