Reference Class Forecasting

Reference class forecasting is a technique for improving predictions across a broad range of problems developed by Kahneman and Tversky. Consider a simple example: You are trying to decide how long it will take to pack your things for a move. One way of doing this might be to imagine all of the things that need to be done, imagining yourself doing them, and then adding up all the times generated by this process. This almost never works. A common technique to fix this is to go through this process and then double or triple the estimate. This sort of works, but we can do better. A process that generates more reliable predictions is to take the average of how long similar plans took in the past, for yourself or others.

Planning by reasoning about each step from scratch is often referred to as the inside view, while reasoning about it based on data gathered from past attempts is often referred to as the outside view. Both forms of reasoning are valuable in problem solving, but most people do too little outside view reasoning. To be more precise, most people anchor on their inside view and then make small adjustments based on outside view data (such as doubling our inside view estimate as above). Best practice is to invert this, use outside view data as your anchor, then make small adjustments based on inside view considerations.

A problem immediately presents itself when trying to apply this in real life. What counts as a valid reference class to draw from? In some cases, like the example above, it is fairly straightforward. In many cases, especially when doing something new, one has a choice about where to gather data from. Do we gather only from data that is extremely similar even though this might not give us very many examples to draw from? Do we gather from many examples that are more loosely defined? Along which dimension are we regarding two examples as "similar"?

These questions don't have precise, rigorous answers. But reference class forecasting tends to improve prediction quality by a lot despite these problems.

Further reading:

http://lesswrong.com/lw/hzu/model_combination_and_adjustment/