Scoreboard: Maby vs. Disease Experts

Keeping score of forecasts is the key to assessing and improving your forecast accuracy. Here we show how it is done.

As the covid-19 pandemic hit the United States, we were invited to submit forecast estimates alongside a CDC-funded panel of experts in public health. Our forecasts have been more accurate than virus experts despite having no training in public health, virology, or epidemiology. We simply follow the same sound forecasting techniques that we teach to our clients.

Below you can see a complete record of our coronavirus forecasts.

Forecast
(10%-90% interval) 		Accuracy Score
(Brier score)
wk Question TOM EXPERTS Outcome TOM EXPERTS
AVERAGE BRIER SCORE
(green is better, red is worse)		 0.217 0.268
5/10 How many cases per day will Washington state average for the week ending June 7? 170
(30-500) 		331
(158-644) 		155 0.289 0.713
5/10 How many confirmed US cases will there be on May 17? (in millions) 1.48
(1.45-1.57) 		1.5
(1.47-1.6) 		1.48 0.062 0.135
5/10 How many deaths will Pennsylvania report as of June 13? (in thousands) 7.6
(5.2-12) 		7.2
(5.2-10) 		6.2 0.214 0.180
5/3 What will the 7 day average new cases be in Texas for the week ending June 13? 1150
(400-3000) 		1350
(750-2300) 		1779 0.323 0.166
5/3 How many confirmed US cases will there be on May 10? (in millions) 1.33
(1.27-1.39) 		1.35
(1.25-1.44) 		1.33 0.066 0.129
5/3 How many US COVID deaths will occur in 2020? (in thousands) 160
(90-800) 		256
(118-1212)
5/3 How many US states/territories will report more new cases for September compared to June? 7
(1-42) 		21
(2-45) 		45 1.088 0.660
4/26 What will the average new daily cases be for Georgia between 5/10 and 5/16 inclusive? 900
(150-5000) 		1044
(579-2292) 		659 0.159 0.173
4/26 How many confirmed US cases will there be on May 3? (in millions) 1.14
(1.09-1.21) 		1.14
(1.07-1.24) 		1.152 0.174 0.195
4/26 When will weekly new US deaths first fall below 5,000 (Sunday to Saturday)? 5/23
(5/9-6/26) 		6/13
(5/16-7/4) 		6/20 0.376 0.158
4/19 How many deaths due to COVID-19 will occur in the US in 2020? (in thousands) 80
(48-750) 		150
(72-517)
4/19 As of May 9, how many COVID-19 related deaths will have occurred in the US? (in thousands) 56
(47-77) 		70
(51-106) 		73 0.420 0.170
4/19 As of May 9, how many COVID-19 related deaths will have occurred in Illinois? (in thousands) 2.4
(1.7-4.8) 		2.8
(1.8-5.2) 		3.3 0.294 0.170
4/19 As of May 9, how many COVID-19 related deaths will have occurred in Louisiana? (in thousands) 1.85
(1.55-3.4) 		2.8
(1.85-5.0) 		2.2 0.129 0.170
4/19 How many confirmed US cases will there be on April 26? (in thousands) 930
(870-990) 		950
(860-1060) 		959 0.222 0.127
4/12 How many US COVID-19 cases will there be on April 19? (in thousands) 755
(710-795) 		780
(690-905) 		751 0.093 0.151
4/12 How many US COVID-19 deaths will occur by May 1? (in thousands) 50
(36-66) 		46
(34-68) 		59 0.111 0.180
4/12 How many states will report more than 1000 deaths by May 1? 12
(9-16) 		9
(4-15) 		14 0.193 0.712
4/12 Which of the next 6 months will see the highest total number of deaths nationwide in the US for COVID-19 illness? April
(April-June) 		April
(March-July) 		April 0.037 0.108
4/12 How many US COVID-19 deaths will occur by June 1? (in thousands) 71
(48-127) 		71
(45-131) 		99 0.178 0.180
4/5 How many US COVID-19 cases will there be on April 12? (in thousands) 520
(470-680) 		670
(510-900) 		551 0.103 0.336
4/5 Which month (March-August) will see the highest total number of hospitalizations nationwide in the US for COVID-19 illness? April
(April-June) 		May
(April-August) 		April 0.043 0.370
3/29 Which month (March-August) will see the highest total number of hospitalizations nationwide in the US for COVID-19 illness? April
(April-June) 		May
(April-August) 		April 0.137 0.344
3/29 How many deaths due to COVID-19 will occur in the US in 2020? (in thousands) 240
(75-1200) 		260
(80-1000)
3/29 Will NY state report fewer than 1,000 new cases on the day of April 28? 28%
(-) 		50%
(-) 		no 0.157 0.500
3/29 How many US COVID-19 cases will there be on April 5? (in thousands) 410
(250-590) 		420
(260-580) 		332 0.123 0.145
3/22 Will US COVID-19 deaths exceed 245,500 by the end of 2020? (percent probability YES) 45%
(-) 		50%
(-)
3/22 Will there will be a second wave of US cases in 2020? (US active cases declines by 50% and subsequently increases by 50% AND 2nd wave must be at least 50% as high) (probability YES) 52%
(-) 		73%
(-)

How were Brier scores calculated for the COVID scoreboard?

When the original question asked by UMass is a binned question then Brier scores are calculated using ordered multinomial scoring. If a question has 4 bins (A, B, C, D) then we will take the average Brier score of 3 binary questions with bins A vs. BCD, AB vs. CD, ABC vs. D.

When a question asks for 10/50/90 percentiles, we have use the percentile forecasts provided by the experts as cut points for a multi-bin question. By definition the experts' forecasts to that multi-bin question are 10% / 40% / 40% / 10%. We have assigned probabilities to those bins consistent with Tom's 10/50/90. This is done as close to real time as possible and posted on twitter.

The table below summarizes all the forecasts that have been made. The forecast column shows forecast probabilities by bin. The bin cut points column indicates where the bin boundaries are.

For example, consider the "WA daily average cases 6/7" question. The cutpoints 158/331/684 were determined by the experts' 10/50/90 intervals. This means that the bins are (A) less than 158; (B) 158 to 331; (C) 331 to 684; (D) more than 684; By construction the experts forecast 10% chance for bins A & D and 40% chance for bins B and C. Tom's 10/50/90 itervals were 30/170/500. Once the experts' intervals were revealed Tom assigned probabilities to the bins A-D consistent with his 10/50/90. This was done as close to real time as possible and recorded on twitter.

scoreboard