In alphabetical order: Johan Almenberg1, Adam Altmejd2, Alexander Bird3, Yiling Chen4, Anna Dreber2, Emma Heikensten2, Magnus Johannesson2, Marcus Munafò5, Thomas Pfeiffer6 and Brad Wilson7
For two weeks (Nov 17-Dec 1) we ran a prediction market on the outcome of the REF for chemistry departments in the UK. The REF will publish an overall quality profile for each department. This can be used to calculate a single number summary score (which the REF itself does not do), which we refer to as the REF score (like a GPA): this simply weights 4* as 4, 3* as 3 etc and calculates the score as a weighted average from the shares of 4, 3 etc.
In the prediction market, participants bet on their beliefs of the REF scores. A prediction market essentially means that individuals can bet on whether an outcome will occur or not, and the market aggregates the information from these bets. If many participants are trading in such a market, market prices will generate a prediction of the outcome, based on the aggregated information of the participants.
After having invited all PhD students and researchers at all UK chemistry departments, we ended up with 17 participants who had the possibility to trade on all 33 departments. The prediction market was run in collaboration with Consensus Point.
In this prediction market, participants were given 10,000 points, the equivalent of £30, which they could use for betting on the different departments. Since REF scores lie between 0 and 4, we created different intervals and let participants bet on these. The intervals were the following: score≤2, 23.50. For all departments, the starting prices for all intervals were the same.
An estimated score is calculated based on the forecasted probability of each interval and its midpoint. For the outer intervals, we used 1.875 and 3.625 as midpoints.
With this method, the prediction market gives us the following ranking of the 33 departments: