The Judgment Deficit - a real-wordliness deficit

I usually don't use my blog to take on or pick apart published pieces - my aim with the blog is to create a diversity of ideas and viewpoints to the reader. There is plenty of intelligent writing in the Web that is thought-provoking and worth bringing to the attention of readers interested in the general ideas of statistics and machine learning. But I came across this learning recently that - I have to admit - caused a fair amount of angst and therefore an urge-to-act. This was the Judgment Deficit by Amar Bhide, a professor of Finance at Tufts University.

The journal article from HBS talks about how machines or computers can make decisions in certain types of situations and human judgment needs to come in at other places. Fair enough. The article then bemoans the recent Great Recession and lays part of the blame on statistical models used in Finance. Specifically the author says
In recent times, though, a new form of centralized control has taken root: mechanistic decision making based on top-down statistical models and algorithms. This has been especially true in finance, where risk models have replaced the judgments of thousands of individual bankers and investors, to disastrous effect.This kind of thinking is not only delusional but also dangerous. (Another part of the article that didn't necessarily get me singing from the rooftops was the lengthy encomium heaped on the economics of Freidrich Hayek, the libertarian economist and the founder of the famous Austrian Economists school. I am still not clear how is that related to the topic at hand.)

The fundamental reason why banks took the risks they took were because there were incentives to do so and there was not enough of an appreciation of the downside. Bankers thought the spiral of rising home-prices, the ability to take the assets off balance sheet and maintain minimal capital reserves, was an unending one and were unable to either spot the inevitable edge of the cliff or were too late to pull back once they spotted it. Also the desire to have these activities as unregulated as possible (to allow free pursuit of profit, or to make 'markets efficient' as Wall Street would argue) led to a number of opacities (about risk) developing in the system which lead to situations where high-schools in Norway were exposed to the collapse of Bear Stearns. So lets not put the blame on top-down statistical models and algorithms. If the alternative that Bhide suggests, having manual underwriters take more of the decisions, were to have happened, I am not sure whether the conclusions reached by these underwriters would have been any different. Apart from a few economists, fund managers and people like Nourini and Taleb (who have made an image of themselves as Cassandras of Doom and therefore have to say anything to maintain that image), nobody - let me say that again - nobody saw this edifice collapsing. No one thought house prices in the US would ever come down. Everyone (human beings and computers alike) were victims of the rear-view mirror bias, i.e. expecting that the future would play out exactly as the past.

So let's go a little bit easy on computers, statistical models and automated decision making.