Belief aggregation, also called risk aggregation, opinion aggregation or probabilistic opinion pooling, is a process in which different probability distributions, produced by different experts, are combined to yield a single probability distribution.
Expert opinions are often uncertain. Rather than saying e.g. "it will rain tomorrow", a weather expert may say "it will rain with probability 70% and be sunny with probability 30%". Such a statement is called a belief. Different experts may have different beliefs; for example, a different weather expert may say "it will rain with probability 60% and be sunny with probability 40%". In other words, each expert has a subjeciive probability distribution over a given set of outcomes.
A belief aggregation rule is a function that takes as input two or more probability distributions over the same set of outcomes, and returns a single probability distribution over the same space.
Documented applications of belief aggregation include:
During COVID-19, the European Academy of Neurology developed an ad-hoc three-round voting method to aggregate expert opinions and reach a consensus.
Common belief aggregation rules include:
Dietrich and List present axiomatic characterizations of each class of rules. They argue that that linear aggregation can be justified “procedurally” but not “epistemically”, while the other two rules can be justified epistemically. Geometric aggregation is justified when the experts' beliefs are based on the same information, and multiplicative aggregation is justified when the experts' beliefs are based on private information.
A belief aggregation rule should arguably satisfy some desirable properties, or axioms:
Most literature on belief aggregation assumes that the experts report their beliefs honestly, as their main goal is to help the decision-maker get to the truth. In practice, experts may have strategic incentives. For example, the FDA uses advisory committees, and there have been controversies due to conflicts of interests within these committees. Therefore, a truthful mechanism for belief aggregation could be useful.
In some settings, it is possible to pay the experts a certain sum of money, depending both on their expressed belief and on the realized outcome. Careful design of the payment function (often called a "scoring rule") can lead to a truthful mechanism. Various truthful scoring rules exist.
In some settings, monetary transfers are not possible. For example, the realized outcome may happen in the far future, or a wrong decision may be catastrophic.
To develop truthful mechanisms, one must make assumptions about the experts' preferences over the set of accepted probability-distributions. If the space of possible preferences is too rich, then strong impossibility results imply that the only truthful mechanism is the dictatorship mechanism (see Gibbard–Satterthwaite theorem).
A useful domain restriction is that the experts have single-peaked preferences. An aggregation rule is called one-dimensional strategyproof (1D-SP) if whenever all experts have single-peaked preferences, and submit their peaks to the aggregation rule, no expert can impose a strictly better aggregated distribution by reporting a false peak. An equivalent property is called uncompromisingness: it says that, if the belief of expert i is smaller than the aggregate distribution, and i changes his report, then the aggregate distribution will be weakly larger; and vice-versa.
Moulin proved a characterization of all 1D-SP rules, as well as the following two characterizations:
Jennings, Laraki, Puppe and Varloot present new characterizations of strategyproof mechanisms with single-peaked preferences.
A further restriction of the single-peaked domain is that agents have single-peaked preferences with L1 metric on the probability density function. That is: for each agent i, there is an "ideal" probability distribution pi, and his utility from a selected probability distribution p* is minus the L1 distance between pi and p*. An aggregation rule is called L1-metric-strategyproof (L1-metric-SP) if whenever all experts have single-peaked preferences with L1 metric, and submit their peaks to the aggregation rule, no expert can impose a strictly better aggregated distribution by reporting a false peak. Several L1-metric-SP aggregation rules were suggested in the context of budget-proposal aggregation:
However, such preferences may not be a good fit for belief aggregation, as they are neutral - they do not distinguish between different outcomes. For example, suppose there are three outcomes, and the expert's belief pi assigns 100% to outcome 1. Then, the L1 metric between pi and "100% outcome 2" is 2, and the L1 metric between pi and "100% outcome 3" is 2 too. The same is true for any neutral metric. This makes sense when 1,2,3 are budget items. However, if these outcomes describe the potential strength of an earthquake in the Richter scale, then the distance between pi to "100% outcome 2" should be much smaller than the distance to "100% outcome 3".
Varloot and Laraki study a different preference domain, in which the outcomes are linearly ordered, and the preferences are single-peaked in the space of cumulative distribution function (cdf). That is: each agent i has an ideal cumulative distribution function ci, and his utility depends negatively on the distance between ci and the accepted distribution c*. They define a new concept called level-strategyproofness (Level-SP), which is relevant when society's decision is based on the question of whether the probability of some event is above or below a given threshold. Level-SP provably implies strategyproofness for a rich class of cdf-single-peaked preferences. They characterize two new aggregation rules:
Other results include:
ANDURIL is a MATLAB toolbox for belief aggregation.
Several books on related topics are available.