Swarm Intelligence, Ant Colony Optimizations – advances in analytic computing

Advances in computing have led to some new and interesting developments in the areas of new modeling techniques. This post is going to give some examples of these kinds of techniques. But before that, a small primer on basic modeling techniques. Most of the more commonly used models are generalized linear models. As the name suggests, these models try to establish a more-or-less linear relationships between what is tried to be predicted and what the inputs are. Ultimately the model fit problem is an optimization problem – an attempt to use a generalized curve to represent the data and while doing so, minimize the gap between the actual data and the approximate representation of the data produced by the model.

Of course, optimization problems present themselves in a number of areas. One is of course model fitting but other applications are in areas like planning and logistics – an example being the ever-popular traveling salesman problem. One of the more recent and interesting techniques in solving optimization problems is through a technique called Ant Colony Optimization (ACO). The optimization is a part of series of more generic AI/ machine learning tools called swarm intelligence. Wikipedia defines swarm intelligence as follows

Swarm intelligence (SI) is the collective behaviour of decentralized, self-organized systems, natural or artificial…. SI systems are typically made up of a population of simple agents or bodies interacting locally with one another and with their environment. The agents follow very simple rules, and although there is no centralized control structure dictating how individual agents should behave, local, and to a certain degree random, interactions between such agents lead to the emergence of "intelligent" global behavior, unknown to the individual agents.

The ACO algorithm tries to mimic the behaviour of ants in search of food. When ants forage for food, every ant involved in the foraging process moves out of the colony in random ways to search for food. When a food source is located, the ant uses the scent trail of its own pheromones to bring the food back to the colony. Other ants begin to then use the trail left behind by the first ant to make further excursions to the food source and bring back food. Also, by the very nature of the pheromone trail (which is a volatile chemical and therefore evaporates after a certain point in time), the tendency of later ants is to follow more recent and fresher trails, which should also be the shortest ones logically speaking.

One of the more interesting business applications has been indeed in the area of material movement, i.e. logistics. The Italian pasta maker Barilla as well as Migros, the Swiss supermarket chain have been using these techniques to optimize their distribution networks and routes. A paper about this technique is available here. It is more technical. A more layman-friendly treatment of the technique appeared recently in the Economist and was also an interesting read.

Animating the data and better "story telling"

One of the challenges with talking about and presenting any analysis about data mining or statistics is that a lay audience is seldom excited by the same things as a more technical audience. A technical audience is as interested in how the answer was reached as much as the answer itself. A non-technical consumer of the same information is probably interested in the implications of the answer as well as the answer itself, with some gut-check to make sure that the process wasn't totally crazy. In other words, they are looking for a story.

Recent trends around the pervasiveness of data and data-driven applications has meant that there is a greater ask from data scientists to tell a compelling "story" to support their analysis. Data scientists need to come up with ways that tell the story behind the data and the projections of the model that may have used the data as input, that are insight generating, that skip some of the unnecessary detail and also paint the various facets of the final solution. And not just tell the final answer. Data animation and data visualization are some of the answers here.

I came across a couple of good examples of such animation recently. Hans Rosling made an interesting presentation about a tool called Gapminder at TED.com. The presentation is here. Gapminder is an organization that makes social, environmental and economic development data from all the countries of the world available and accessible to all, for free. The visualization tool at Gapminder called Gapminder World shows ways in which this data can be animated and made come alive for the non-technical consumer in illuminating and exciting ways.

Rosling made another trailer presentation recently from a BBC 4 program promo called "The Joy Of Stats". (Link is here if the embed doesn't work.). One hopes that this program airs sometime in the US. It is due to air on Dec 7 and 8 in the UK. Any UK readers of the blog are encouraged to go and check the program and share what they felt about it. The content of the program (Link and timings here) sounds interesting enough for me to at least contemplate taking a flight to London and catching the program on the Beeb.

The quintessential Greek (financial) tragedy

For the last 6 month, the travails of highly indebted countries in the European Union and Greece in particular has been the source of considerable turmoil in the financial system. In addition to increasing the cost of borrowing for Greece and other countries in a similar situation like Spain, Portugal, Ireland and Italy, the other impact has been to increase the overall systemic risk and push the world economy back into the 2008 depths.


The Greece story is particularly fascinating. Unlike other countries where banks made ruinous bets and had their capital wiped out, impacting lending and slowing down economic activity, the banks had no role to play in Greece. Instead it was the systemic lack of fiscal discipline, lack of enforcement of basic property and taxation principles and a proliferation of special interest driven that causes Greece to be on a slippery slope to sovereign bankruptcy and default. The inimitable Michael Lewis has written a highly entertaining but also illuminating piece on why this came to happen. Link is here.


Now what is very interesting and scary is that many of the ills mentioned here is present in many countries around the world. Talk about the aversion towards taxes, the large scale tax evasion, the rampant bribery and corruption in government circles. Seems scarily familiar to people from India and other developing countries. What do you think causes Greece to fail in such spectacular fashion (well, if it hasn't already failed, this article should convince you to "short" Greek debt).

Sensational ... but true

Photograph by Terry McCombs

My writings in this space are usually somewhat academic and what I think is an objective view point, whenever I stray from highly academic topics. A recent article from the Rolling Stone magazine, though, caught my eye. The writer talks about the systematic way in which Wall Street banks touting their financial engineering expertise have expanded their influence into small governments in America. The banks have peddled products that have invariably resulted in huge amount of financial distress for these agencies. See link here.

The phenomenon of bankers behaving badly with small governments or for that matter even with larger ones (state pension funds) is nothing new. The 1994 bankrupting of Orange County by Robert Citron (Citron? for Orange Country?), the county treasurer, is a well-known story. See a really good article about this failure here. Another notable example from the same period is Proctor & Gamble's dalliance with derivatives that resulted in a lot of grief for themselves as well as for Banker's Trust, their investment banking advisers. The human psyche seems particularly frail and susceptible to smooth talking operators, talking interesting numbers and displaying other forms of spreadsheet gadgetry, and promising the moon in return for money. In addition to the banks' rapacity, the people at the customer end - that sought to invest in little understood financial instruments, where the risk of the counterparty is bounded but your own downside is infinite - are as much to blame. Not for the lack of financial savvy, but for getting into a situation where such financial gimmickry needed to be resorted to in the first place.

The primary problem here is the particular weakness of small government bodies to be reckless about spending during good times. In an attempt to do something big and important for their constituents (ascribing the best motives), governmental bodies take on big projects when the economic cycle is positive and tax revenues are abundant. They take on big loans which need servicing even when things go bad - when tax revenues decline or interest rates rise or whatever. And then these agencies find themselves strapped for money and start to resort to financial gimmickry. And fall into the arms of the Wall Street firms.

Reminds me of the famous Roald Dahl story about the old man who has a priceless painting tattooed on his back. And who goes away with a smooth talking stranger who promises to keep him happy for the rest of his life, only if the old man displayed his painting to the stranger's guests at his hotel. Needless to say, but in a few weeks, the painting appears sans the old man in a famous art gallery.

Facebook's "revenue model"

A really interesting article from Businessweek on the purported revenue model behind Facebook. One of the more insightful articles I have read recently.

Statistical models and ivory towers

Increasingly, business analytics and the use of advanced statistics in business decision making is making big surges. Companies from WalMart to FedEx to Netflix have demonstrated how they can build a sustainable business model on the foundations of good data and solid analysis of that data. To analyze the data, people with both the right statistical background and training as well as the required business acumen are critical. In other words, smart people. And this is usually one of the barriers to organizations making the transition from the pre-analytics to the post-analytics space.

Ever so often, the people who push analytics within an organization come from an angle of intellectual superiority. "I can do math better than you and therefore I am right and better than you" is the mindset that many such practitioners bring to the field. This often results in resistance and sometimes, downright hostility to what the "statistical ones" are recommending, from the rest of the organization. Statistical practitioners often end up plowing a lonely furrow in organizations. And then one day, when the implicit sponsorship that got them into that position goes away, so follow the statistical modelers out of the organization. The feeling when they leave is one of profound disappointment and disillusionment on the side of the modelers, and profound relief and also some good old schadenfreude on the side of the old organization hands. How can this situation be averted? How can people who are obviously so intelligent and well-educated prevent making fools of themselves because they failed to fit into an organization?

A few pieces of advice:
1. Be there to "solve the problem" vs "showcase your smarts"
It is important to keep in mind why smart people are hired by organizations. It is usually to solve some business problem or the other. It is not because the organization suddenly discovered that they needed show ponies to come out and parade their smarts. So the first advice to the smart ones is to focus on fixing organizational challenges, i.e. focus on what they are hired for and build their credibility. Once credibility is built up, it becomes infinitely easier to take on work that is more intellectually stimulating and challenging.

2. Simplify your communication around the solution
Smart people often have the ability to get into really deep thinking about the work that they are involved in. Deep thinking indeed is required to fix many of the more difficult problems that companies and society is faced with. However deep thinking around communications is counter-productive. Human beings are simple creatures and usually favour clean narratives over complex ones. Keep the communication around the solution simple and crisp - you may need to get rid of some of the fancy footwork to get there but the trade-off is usually worth it.

3. Be open to idea "give and take"
Finally, approach idea sharing with positive intent. Ideas usually get better when they are critiqued by other people. Valuable perspectives come to light and unrecognized (by the idea creator) weaknesses are called out. Smart people tend to have a bias towards thinking "my way or the highway". This not only prevents ideas from realizing their full potential but also destroy the buy-in that is required from stakeholders. Buy-in is the oxygen that ideas need to survive and grow and developing the political savvy and getting that buy-in is always critical.

The Judgment Deficit - a real-wordliness deficit

I usually don't use my blog to take on or pick apart published pieces - my aim with the blog is to create a diversity of ideas and viewpoints to the reader. There is plenty of intelligent writing in the Web that is thought-provoking and worth bringing to the attention of readers interested in the general ideas of statistics and machine learning. But I came across this learning recently that - I have to admit - caused a fair amount of angst and therefore an urge-to-act. This was the Judgment Deficit by Amar Bhide, a professor of Finance at Tufts University.

The journal article from HBS talks about how machines or computers can make decisions in certain types of situations and human judgment needs to come in at other places. Fair enough. The article then bemoans the recent Great Recession and lays part of the blame on statistical models used in Finance. Specifically the author says
In recent times, though, a new form of centralized control has taken root: mechanistic decision making based on top-down statistical models and algorithms. This has been especially true in finance, where risk models have replaced the judgments of thousands of individual bankers and investors, to disastrous effect.This kind of thinking is not only delusional but also dangerous. (Another part of the article that didn't necessarily get me singing from the rooftops was the lengthy encomium heaped on the economics of Freidrich Hayek, the libertarian economist and the founder of the famous Austrian Economists school. I am still not clear how is that related to the topic at hand.)

The fundamental reason why banks took the risks they took were because there were incentives to do so and there was not enough of an appreciation of the downside. Bankers thought the spiral of rising home-prices, the ability to take the assets off balance sheet and maintain minimal capital reserves, was an unending one and were unable to either spot the inevitable edge of the cliff or were too late to pull back once they spotted it. Also the desire to have these activities as unregulated as possible (to allow free pursuit of profit, or to make 'markets efficient' as Wall Street would argue) led to a number of opacities (about risk) developing in the system which lead to situations where high-schools in Norway were exposed to the collapse of Bear Stearns. So lets not put the blame on top-down statistical models and algorithms. If the alternative that Bhide suggests, having manual underwriters take more of the decisions, were to have happened, I am not sure whether the conclusions reached by these underwriters would have been any different. Apart from a few economists, fund managers and people like Nourini and Taleb (who have made an image of themselves as Cassandras of Doom and therefore have to say anything to maintain that image), nobody - let me say that again - nobody saw this edifice collapsing. No one thought house prices in the US would ever come down. Everyone (human beings and computers alike) were victims of the rear-view mirror bias, i.e. expecting that the future would play out exactly as the past.

So let's go a little bit easy on computers, statistical models and automated decision making.