|
Posted on March 8, 2016 @ 10:39:00 AM by Paul Meagher
In my last blog I started discussing the book Superforecasters: The Art and Science of Prediction (2016) by Philip Tetlock and Dan Gardner. I suggested that financial forecasting is a useful arena in which to hone forecasting skills and I used the example of forecasting my
book expenses for 2016. I estimated that I would purchase 52 books (average of 1 per week) and each book would cost $30 so my overall projected expenses for books in 2016 was $1,560.
It turns out that when I actually tally up all the books I purchased since the beginning of the year until Mar 1, 2016, sum the cost for all of them
and add taxes the amount is $583.43 (I don't generally incur any shipping costs). I purchased 20 books in that period. Average cost per book was $29.17 (which was very close to my
estimate). If I assume that I will spend the same amount over the next 10 months then my forecasted book expenses would be $3500.57. The
difference between my initial estimate of $1,560 and this estimate of $3500.57 is $1940.57. We have quite a discrepancy here.
When you make a forecast that forecast should not be written in stone. It is based upon the best information you had available at the time.
You learn new information and the world changes so you have to adjust your forecasts to incorporate this new information. When superforecasters update their forecasts the change from the previous forecast is generally not a big shift although it can happen. The information up to that point still has some weight in determining what the current forecast should be. Forecasters need to be wary of overreacting to new information by making large changes to their forecast right away.
Likewise in light of the new information that my book expenses could be $3500.57 I have to decide how to incorporate this new information
into my current forecast of $1,560. Because my estimate of the cost per book was quite accurate ($30) the question boils down to whether
I will end up purchasing 116 books instead of the 52 I estimated. Even though I like books I can't see me doubling my current rate of
book buying. I don't expect to keep buying at this rate during the spring/summer as I won't have as much time for reading. So I am inclined to remain close to my original forecast but perhaps bump it up a bit to take into account the hard data I have on how much I spent so far.
Financial forecasting is subject to events happening in the world but it is also subject to a policy decision that will control costs. My
policy decision is to purchase at the rate of 1 book a week however I will also sometimes buy books more impulsively if I'm in a bookstore,
or, as happened last Saturday a local book author was at a seed buying event and I purchased her new Permaculture book. So my model of book purchasing consists of a policy component involving 1 book a week and another "random" component which I'll simply assume amounts to 1
book a month over and above my policy. This will generate a forecast of 64 books this per year at $30 per book with is $1920. So my
forecasted 2016 Book Expenses has moved from $1560 to $1920 as a result of new information about my actual book purchasing costs to date.
I could wait until the end of the year to see how close my forecasted book expenses are to my actual book expenses, but why wait until then?
I might want to check in after 6 months and see where I stand then and adjust my forecast accordingly. After six months my expenses should
be half of $1920 or $960. So I'll check in again at 6 months and see if my expenses are close to this amount. Superforecasters regularly
update their forecasts and will also often do a post-mortem when examining forecast accuracy to figure out what they did right or wrong.
Incorporating feedback in this way helps to improve future forecasting in that domain.
Another way to make forecasts instead of simple point estimates as I have done is to forecast that my book costs will fall within some interval
with a certain probability. So I might say that in 6 months my book expenses will fall within +- 60 dollars of $960 with a probability
of 80%. The two ways I can improve upon my future forecasts is to 1) narrow my range (to +- 30 dollars) and 2) increase my estimate of its
probability (to 90%). One method we can use to score such forecasts is quadratic scoring which penalizes you more for incorrect estimates
that are assigned a high probability (90%) of being true compared to a lower probability of being true (60%). I'll leave the discussion of the math used for quadratic scoring for my next blog.
The purpose of this blog was to discuss the idea that being a better forecaster involves updating your forecast as you assimilate new information rather than just making a forecast and waiting until the forecast date to see if it is correct. Superforecasters update their forecasts regularly, they generally don't overreact by making big shifts from their previous forecasts. They analyze what went right or wrong when a forecast is checked against actual numbers so they can use this feedback to improve their future forecasts. It is hard to assign a probability to a point estimate so we introduced the idea of assigning a probability that the forecasted number would fall within some range. In my next blog we will look at quadratic scoring (or Brier scoring) used to evaluate how good these forecasts are.
|
Animatronics and Humanoid Robots
