This page explains in the details behind how Brier scores are calculated for the COVID scoreboard.
When the original question asked by UMass is a binned question then Brier scores are calculated using ordered multinomial scoring. If a question has 4 bins (A, B, C, D) then we will take the average Brier score of 3 binary questions with bins A vs. BCD, AB vs. CD, ABC vs. D.
When a question asks for 10/50/90 percentiles, we have use the percentile forecasts provided by the experts as cut points for a multi-bin question. By definition the experts' forecasts to that multi-bin question are 10% / 40% / 40% / 10%. We have assigned probabilities to those bins consistent with Tom's 10/50/90. This is done as close to real time as possible and posted on twitter.
The table below summarizes all the forecasts that have been made. The forecast column shows forecast probabilities by bin. The bin cut points column indicates where the bin boundaries are.
For example, consider the "WA daily average cases 6/7" question. The cutpoints 158/331/684 were determined by the experts' 10/50/90 intervals. This means that the bins are (A) less than 158; (B) 158 to 331; (C) 331 to 684; (D) more than 684; By construction the experts forecast 10% chance for bins A & D and 40% chance for bins B and C. Tom's 10/50/90 itervals were 30/170/500. Once the experts' intervals were revealed Tom assigned probabilities to the bins A-D consistent with his 10/50/90. This was done as close to real time as possible and recorded on twitter.
