Statistical models and ivory towers

Increasingly, business analytics and the use of advanced statistics in business decision making is making big surges. Companies from WalMart to FedEx to Netflix have demonstrated how they can build a sustainable business model on the foundations of good data and solid analysis of that data. To analyze the data, people with both the right statistical background and training as well as the required business acumen are critical. In other words, smart people. And this is usually one of the barriers to organizations making the transition from the pre-analytics to the post-analytics space.

Ever so often, the people who push analytics within an organization come from an angle of intellectual superiority. "I can do math better than you and therefore I am right and better than you" is the mindset that many such practitioners bring to the field. This often results in resistance and sometimes, downright hostility to what the "statistical ones" are recommending, from the rest of the organization. Statistical practitioners often end up plowing a lonely furrow in organizations. And then one day, when the implicit sponsorship that got them into that position goes away, so follow the statistical modelers out of the organization. The feeling when they leave is one of profound disappointment and disillusionment on the side of the modelers, and profound relief and also some good old schadenfreude on the side of the old organization hands. How can this situation be averted? How can people who are obviously so intelligent and well-educated prevent making fools of themselves because they failed to fit into an organization?

A few pieces of advice:
1. Be there to "solve the problem" vs "showcase your smarts"
It is important to keep in mind why smart people are hired by organizations. It is usually to solve some business problem or the other. It is not because the organization suddenly discovered that they needed show ponies to come out and parade their smarts. So the first advice to the smart ones is to focus on fixing organizational challenges, i.e. focus on what they are hired for and build their credibility. Once credibility is built up, it becomes infinitely easier to take on work that is more intellectually stimulating and challenging.

2. Simplify your communication around the solution
Smart people often have the ability to get into really deep thinking about the work that they are involved in. Deep thinking indeed is required to fix many of the more difficult problems that companies and society is faced with. However deep thinking around communications is counter-productive. Human beings are simple creatures and usually favour clean narratives over complex ones. Keep the communication around the solution simple and crisp - you may need to get rid of some of the fancy footwork to get there but the trade-off is usually worth it.

3. Be open to idea "give and take"
Finally, approach idea sharing with positive intent. Ideas usually get better when they are critiqued by other people. Valuable perspectives come to light and unrecognized (by the idea creator) weaknesses are called out. Smart people tend to have a bias towards thinking "my way or the highway". This not only prevents ideas from realizing their full potential but also destroy the buy-in that is required from stakeholders. Buy-in is the oxygen that ideas need to survive and grow and developing the political savvy and getting that buy-in is always critical.

It's been a long time

It's been long since I shared my views on my blog. Excuses are multiple, but none is particularly credible.

But I think this is a great time to start again. In the last 2 years, the world has been painfully exposed to the fallibility of models. From being the engines of modern finance and the economy at large, models have gone to becoming the reason #1 for the economy's collapse. Even worst, model builders have become a bit of a laughing stock in the post-Wall Street society.

Here's an attempt to set models and statistics to back where they belong. So lets see how it goes this time around.

Quantifying the tails - Another interesting book

One of my readers Rusen pointed to another very interesting (if somewhat specialized) book which deals with quantifying extreme or once-in-a-hundred-years events. The name of this book is "Fooled by Randomness: The Hidden Role of Chance in Life and in the Markets" by Nassim Nicholas Taleb.

Taleb talks about events that occur at the extreme end of a probability distribution, where we think our models work but they models break down (because of a number of reasons). Taleb presents a perspective around where we should depend on models and where our intuition should tell us not to trust the models.

If readers want a preview of what's there in the book, check out this website www.fooledbyrandomness.com. (Taleb is releasing a second book this month called "The Black Swan: the impact of the highly improbable".)

Some interesting books on risk and probability

My holiday reading in India is two books I have been planning to read for a while now, but never really found the time.

One is "Against the Gods: The Remarkable Story of Risk" by Peter Bernstein. People who have read Peter Bernstein will know what to expect. Very detailed coverage of fairly arcane concepts without getting into too many technicalities. A good jumping board to get into the actual academic papers. (Something I love doing is to pick these books and then search for the bibliography, esp. academic papers. Very interesting exercise and highly encouraged for those amongst us who want to know a little more.). I should post a review of the book in the next few weeks. For those who want to read another quality book from Peter Bernstein, try "Capital Ideas".

Second is a book called "Chances are: Adventures in Probability" by Kaplan and Kaplan. I am midway through this book. Again very informative and entertaining. The underpinnings of insurance is particularly interesting. (Bayes is coming up in the next chapter. I am super excited!)

A third book I really enjoyed was "When Genius Failed" by Roger Lowenstein. Hope to start a discussion about the book very soon, as I see very immediate application with what LTCM went through and my current line of work. (No, I don't work with a hedge fund.)

Even while I am adding stuff about these books, do try and grab them from the local library or better still, own them. They will give make for pleasurable reading for many many years, I promise. For those living in Fairfax County in Virginia, USA, the library system has all of these books.

Prolog

If you have come to the Statistics Exchange, I am guessing you are someone facinated by Statistics and intrigued by the idea of what an exchange means.

This blog is just coming up but let me give you a preview of what you can expect. This is a place where people can get together and share statistical insights and ideas purely from a practitioner standpoint. Don't expect to see too much of theory here, this space is purely about interesting applications within the space. I am just getting started here and in the middle of a vacation, which is a perfect time to step back and do something like this. My regular job is full of statistical modeling, so it will be exciting to learn things from work and apply them here. And vice versa. Hoping to put some interesting stuff here pretty soon.