The Plight of the Forecasters

Yet again the Indian Meteorological Department (IMD) has confessed– that its forecast (6% above LPA – Long Period Average) of above average monsoon in India may not be met. Understandably this will again fuel a debate on the utility of weather forecasts given the frequency with which IMD and other agencies are unable to call important weather developments like cyclone, rainfall pattern, intensity and duration of heatwave/coldwave etc.

Now, we all make forecasts – in selecting a career path, deciding on a life partner, buying a house, making an investment, relocating to some other city etc. We take these decisions based on our expectations, i.e forecasts, regarding the future.

On the other hand there are bigger developments, relevant for the society on the whole, like capital market movements, elections, diplomatic events, outbreak of a disease, weather patterns etc where experts seem better placed to give advance notice so that the general population is better prepared for that development.

Forecast and folly

Not surprisingly we come across experts who dish out forecasts on every topic under the Sun via various media channels on a daily basis. However if one thinks of forecasts – on weather, stock markets, elections, war etc- one realizes that most experts are woefully short of expectations. Not surprisingly, in his research on this topic over a twenty year period the Wharton professor Philip Tetlock found that an average expert did little better than blind guesses on many political and economic forecasts.

Poor quality of forecasting is to some extent caused by low demand – by consumers of forecasts- for evidence of precision. When there is no measurement metric to test the accuracy of forecasts in stock prices or in election results the forecaster does not have enough incentive to revise his forecast after further focused work. Interestingly, consumers of forecasts are often attracted just by the conviction of the forecaster and the attractiveness of the story. Unfortunately, compelling story and conviction are poor indicators of forecast quality. Apart from this, forecasts are also marred by overconfidence, single directional approach, excess optimism etc.

Ten Commandments for aspiring superforecasters

Ability to forecast is an art (and science) that can be developed by deep, deliberative practice. So what should forecasters do to develop this ability? Philip Tetlock, in his book ” Superforecasting: The Art and Science of Prediction ” has laid this out succinctly.

  1. Triage – Don’t waste time on too easy or impenetrable questions. Here two basic errors could be either the failure to predict the predictable, or time wastage in trying to predict the unpredictable.
  2. Break seemingly intractable problems into tractable sub-problems – Remarkably good probability estimates often arise from remarkably crude assumptions and guesstimates.
  3. Strike the right balance between inside and outside views – Superforecasters know that there is nothing new under the Sun. Nothing is 100% unique. They use this fact to pose outside view questions – how often do things of this sort happen in situations of this sort? Inside view evidence, which is more detailed and direct, has to be super imposed after this.
  4. Strike the right balance in under and over reacting to evidence – Savvy forecasters learn to ferret out telltale views before others. They snoop for non obvious lead indicators and tend to be incremental belief updaters. At the same time Super forecasters also know how to move their probability estimates fast in response to diagnostic signals. Thus a good forecaster must be good at making small, measured revisions but has to be prepared to make large or even directional changes to his forecasts if new evidence requires so.
  5. Look for the clashing casual forces at work in each problem – For every good argument there is typically a counter argument that is at least worth acknowledging. Synthesis of various viewpoints is important.
  6. Strive to distinguish as many degrees of doubt as the problem permits – The more degrees of uncertainty one can distinguish the better forecaster one is likely to be. Translating vague hunches into numeric probabilities can be developed as a habit with patience and practice.
  7. Strike the right balance between over and under confidence, between prudence and decisiveness – Good forecasters routinely manage trade-offs between the need to take decisive stands and the need to qualify their stands. Long term accuracy requires getting good scores on both calibrations as well as resolution – which requires moving beyond blame game. Good forecasters have to find creating ways to tame down both types of forecasting errors – misses, as well as false alarms.
  8. Look for the errors behind your mistakes but beware of hindsight bias – Don’t justify or look for excuses for your failures. Own them. Conduct post mortems – even for successes. Don’t learn too little or too much from your mistakes. Not all successes imply that the reasoning was right. It’s important to get the process right.
  9. Bring out the best in others and let others bring out the best in you
  10. Master the error – Just like one can’t learn riding a bicycle by reading a physics book one can’t become a good forecaster by reading training manuals.
  11. Don’t treat commandments as commandments – Two cases are never exactly the same. So be prepared to modify ground rules as per the situation.

On the monsoon forecasts 2016

Weather is one of the more difficult things to forecast. Especially, as we go farther out into future forecast quality deteriorates. Interestingly, weathermen are amongst the most humble forecasters – more so versus stock market experts or political analysts, perhaps due to the ease of measurability of their forecasts. They are quick to admit mistakes and perhaps that explains the meaningful improvement in quality of weather forecasts over last 3-4 decades. Finally, in this instance even if monsoon turns out to be 4-5% lower than LPA it won’t harm the prospects  of Indian economy much since a) 80% area in India is receiving normal rainfall and , b) Kharif sowing has already been done satisfactorily.