Tips for data mining - Part 3 out of 4

I had written two pieces early on in the year about data mining tips. These talked about the first four tips to keep in mind while undertaking any data-mining project.
1. Define the problem and the design of the solution
2. Establish how the tool you are building is going to be used
3. Frame the approach before jumping to the actual technical solution
4. Understand the data

The links for the first two parts can be found here and here. Now let me talk about the fifth and sixth parts on what I have learned.

5. Beware the "hammer looking for a nail"
This lesson basically recommends that you make sure you are using the appropriate complexity and sophistication of the analytical solution for the problem at hand. It is very easy to get excited about any one analytical solution and try to apply it to every single problem that you come across. But approaching a business problem like a "hammer looking for a nail" creates a set of issues. One, application of the technique becomes more important that understanding the problem. When that happens, the desire to implement the technique successfully becomes more important than solving the problem specifically. Two, the solution can sometimes reach a level of complexity that the problem does not really need - on several occasions, simple solutions work the best.

6. Validate your solution
One of the most common mistakes that a data miner can make when confronted with a problem is to produce an overfit model. An overfit model is an overspecified model - many of the relationships between the predictor and target variables implied by the model are not real. They are an artifact of the dataset used to build the model. The problem with overfit models is that they tend to fail spectacularly when applied to a different situation. Therefore, it is crucial to do out-of-sample validation of the model. If the model does not do a good job validating on the validation sample, it usually means an overspecified model. (Holdouts from the original build dataset - the typical two-third/ one-third breakup between the build and validation datasets - don't really result in an independent validation.) The model might need to be simplified. One way to do it is to examine all the relationships between the predictor variables and the target variable and make sure they are sensible and believable. Another way of simplification is to make sure only linear relationships are maintained in the model. To be clear, this is often a gross oversimplification - but it is sometimes better than an overspecified model that is unusable.

So this was tip #5 and 6. I will soon close out with the last two tips. Thanks for the patience with my slow pace of writing on this. This is the link to Tom Breuer's tips on building predictive models. A lot of good ideas here as well.

Principal Components - the math behind it

A really delightful tutorial on the mathematical basis for Principal Components Analysis. It really clarified a lot of the basics for me.

Interesting uses of Principal Components Analysis

I had shared a link on Principal Components Analysis a while back and have had the opportunity to revisit this space, or rather visit it in a professional capacity recently.

As a part of my interest, I came across a few interesting links on this subject - one of the better tutorials is here. The primary purpose behind PCA is dimensionality reduction to make analysis more efficient. Typical applications have been in the area of image processing, though of late, there has been a lot of interest in applying these techniques for micro-array data.

Some of the historical applications of PCA has been in the field of statistical process control or SPC. The genesis of the application came from the chemical industry, and the early practitioners were interestingly known as chemometricians. The aim here was to model plant yield as a function of its input parameters. The input parameters were typically the temperatures and pressures at different points in the reactor vessel and also recorded at different points in time. Since the plant operator has control over some of these parameters, these can be varied in order to improve the plant yield.

The sheer complexity of the data involved here is one complication. When processes have hundreds on inputs (temperature, pressure, gradients, energy released, moisture content - all captured by hundreds of sensors embedded within the reactor), it becomes difficult to build any meaningful models - given the limited number of observations available. What is helpful is that many of these input variables are highly correlated. The temperature at the entry point of reactor feed is going to be obviously correlated to the temperature at the center of the reactor vessel. PCA can be used to reduce the dimensionality of the inputs and model the outputs as a function of the principal components rather than the input variables. Principal components by definition is simply reducing the hundreds of correlated inputs into a few principal components (typically 3 or 4) which are a linear combination of these raw inputs. The other application here is the monitoring of these reactions. When the operator runs different reactions with different input parameters, it is important to identify 'outliers'. Places where the inputs have been so far away from norms that the outputs need to be appropriately flagged or in some cases, totally discarded. Some more details on the application of these techniques can be found here. The link goes to a paper on Principal Component Techniques by Robert Rodriguez from SAS.

These applications can be extended to other areas as well. In consumer behaviour modeling, PCA can be used to reduce the hundreds of different inputs about a consumer to the essential principal components and these can then be used to simplify the modeling and monitoring processes.

The Heritage Prize

The latest in data mining competitions is the Heritage Prize. If you haven't heard about the prize before, it is a competition to bring predictive analytics to the health-care business. The reward: a cool $3 million. This is the next phase of predictive analytics - when corporates are willing to pay good money for great analytical work.

Tips for data mining - Part 2 out of many

Writing after a long time on the blog. Blame it on regular writer's cramp - a marked reluctance and inertia of sorts to put pen to paper, or rather fingers to keyboard.

My last post introduced the idea of defining the problem as the first step for any data mining exercise aiming to achieve success. This is the ability to state the problem you are trying to solve in terms of business outcomes that are measurable. After that comes the step of envisioning the solution and expressing it in a really simple form. The aim should be to create a path from input to output - the output being a set of decisions that will ultimately result in the measurable business outcomes we mentioned above. The next step involves establishing how the developed solution would be used in the real world. Not doing this early enough or with enough clarity could result in the creation of a library curio. Defining how the solution will be used will also point to other needs such as training the users on the right way to use the solution, the expected skills from the end users and so on.

In this post, we will discuss the third and fourth steps. These are
3. Frame the approach before jumping to the actual technical solution
4. Understand the data

Frame the approach before jumping to the actual technical solution. Once the business problem has been defined, it is tempting to point the closest tool at hand at the data and starting to hack away. But often times, the most obvious answer is not necessarily the right answer. It is valuable to construct the nuts and bolts of the approach to get to the solution on a whiteboard or sheet of paper before getting started. Taking the example of some recent text-mining work I have been involved in, one of the important steps was to create an industry-specific lexicon or dictionary. While creating a comprehensive dictionary is often tedious and dull work, this step is an important building block for any data mining effort and hence deserves the upfront attention. We couldn't have seen the value of this step, but for the exercise of comprehensively thinking through the solution. This is also the place where prototyping using sandbox tools like Excel or JMP (the "lite" statistical software from the SAS stable) becomes extremely valuable. Framing this approach in detail allows the data miner to budget for all the small steps along the way that are critical for a successful solution. It also enables putting something tangible in front of decision makers and stakeholders which can be invaluable in getting their buy-in and sponsorship for the solution.

Understand the data. This is such an obvious step that it has almost become a cliche; having said that, incomplete understanding of the data continues to be the reason why the greatest number of data mining projects falter in attempting to fulfill their potential and solve the business goal. Some of the data checks like data distributions, variable attributes like mean, standard deviations, missing rates are quite obvious but I want to call out a couple of critical steps here that might be somewhat non-obvious. The first is to focus extensively on data visualization or exploratory data analysis. In the blog, I have written a few pieces before on data visualization which can be found here. Another good example of this type of visualization is from the Junk Charts blog. The second is to track data lineage - in other words, where did the data come from and how was it gathered. Also is it going to gathered in the same way going forward. This step is important in understanding whether there have been biases in the historical data. There could be coverage bias or responder bias, where people are invited or requested to provide information. In both these cases, the analytical reads are usually specific to the data collected and cannot be easily extrapolated to non-responders or people outside the coverage of the historical data.

This covers the background work that needs to take place before the solution build can be taken up in earnest. In the next few posts, I will share some thoughts on the things to keep in mind while building out the actual data mining solution.

Tips for data mining - part 1 out of many!

Having spent a good part of the last four years in mining data and working in general in the business/ predictive analytics areas, I thought I'd take a step back and summarize some of my lessons learnt through data mining. I was inspired to do this based on a very revealing article by Tom Breur, principal at XLT Consulting. More on Tom and his writings later.

So what have I worked on these past years? As a management consultant in my previous life, working with data and tons of it was a given. Using data extensively, building business analytics models which aim to replicate real world processes, establishing objective criteria to take decisions is all bread-and-butter ways of problem solving in the consulting world. In fact, I'd go as far to say that consultants seriously suffer from a lack of self-confidence when they consider out of the box solutions that do not include any/ all of the above. My consulting experience was mainly in the area of consumer goods distribution and marketing. Then moved to retail financial services. The main area of experience there has been in credit risk modeling and consumer cash flow modeling. Modeling how much of these events (credit risk, cash flow) are driven by internal factors and how much by exogenous occurences. Also modeling consumer response to marketing products. A recent foray has been into text mining unstructured responses from applicants.

So what have I learnt? I will try and summarize in a few posts, with potential reader fatigue in due consideration.At a summary level, these are the eight steps to data mining salvation.

1. Define the problem and the design of the solution

2. Establish how the tool you are building is going to be used

3. Frame the approach before jumping to the actual technical solution
4. Understand the data
5. Beware the "hammer looking for a nail"
6. Validate your solution
7. Beware the "smoking gun"
8. Establish the value upside and generate buy-in

I will tackle each of these steps in some detail now.

1. Define the problem and the design of the solution
This is the first step. Define the problem that you are really trying to solve. The key here to defining the problem well is to frame the solution in terms of business outcomes.

Complete the following sentence. "Solving this problem will lead to x% increase in sales, y% decrease in costs, a multiplier of efficiency and speed by z" etc. If this sentence does not flow easily, then I am afraid you have not spent the time defining the overall problem well enough.

Understand the context surrounding the problem, why it has been difficult to crack over the years, where is the data going to come from, where has the data come in the past and are there going to be changes to how it will be available in the future? Speak to others who have taken a crack at this problem and their view on where the constraints lie.

Once you have defined the problem well enough, envision what the solution is going to look like. What are the parts of the solution that are pure process excellence, where do you need advanced analytics, so that the place for the analytical piece of the solution (something that the reader of this blog is primarily interested in) is clearly established. The aim should be to create a simple block diagram on how one goes from input (usually data) to output (ideally, a set of decisions) and what are all the pieces that come in between.

2. Establish how the tool you are building is going to be used
In the previous step, the problem has been defined and the solution has been scoped at a high level. Then the next step is to put some detail into how the analytical solution is going to be actually used. Will the model be used mainly for understanding purposes or for doing exploratory analysis? And therefore the results of the analysis implemented using some simple decisioning rules or heuristics. Or is the desire or the plan to use the model in the "live" production environment for decision making? It is important to get good answers or at least good likely answers to all of these questions because they play a very important role in determining the actual tools that will be used to build the solution, the checks and audits that need to be put in the overall system of decision making, the process of overrides, the infrastructure and technology needed to make the solution effective and so on. Also the people aspect needs to be considered at this step. The use-conditions of the tool will determine the type of user training that needs to be provided, also the skills of the end-users that needs to be ensured and how much of the skills can be imparted by on-the-job training vs what skills are entry conditions into the job.

More about all of this in subsequent posts.

The Joy of Stats is finally online

My first post of 2011. I have been writing this blog for nearly two years now and am happy to keep having the energy and the enthusiasm to keep at it. Like I have said earlier on why I blog, this is way for me to keep abreast of the latest development in the fields of data mining, analytics and visualization.

The Joy of Stats program was aired in BBC4 in December 2010. Now the video is available on Hans Rosling's Gapminder website. This was the program from which the data visualization examples used for mapping San Francisco crimes, the graphics made by Florence Nightingale and the Gapminder visualization of the economic and demographic statistics of different countries over the last 200 years are highlighted.

Another example of nifty graphics. The New York Times has been a trendsetter in putting up very clever and informative graphics supporting its news stories. Amanda Cox of the NYT graphics department did a presentation recently on some of the examples that the NYT has used in its print as well as its online media. This is a long presentation but worth sitting through.

Hopefully you will enjoy both these presentations!