Friday, July 3, 2009

Best Coffee Survey and a research methodology question

A recent Zagat survey rated the best coffee in the US. The best coffee rating went (expectedly, I guess) to Starbucks. Even though I have had better coffee at other places, I guess Starbucks combines great coffee with ubiquitous presence and therefore ends up getting the top rating. Now, I think Starbucks coffee is good and the baristas are extremely friendly, but in terms of pure coffee flavour, I would rate Panera's Hazelnut coffee higher. Also some of the Kona coffees that you find at places like WaWa are also really good. Any kind of place serving Jamaica's Blue Mountain coffee will obviously be great. So what makes Starbucks special? Are there other factors at play beyond the pure taste of the coffee.

One hypothesis is that the national-level presence of Starbucks could be contributing towards the voting going for Starbucks. In places where Starbucks has to compete with other chains like Peets (San Francisco) and Dunkin Donuts (New England), comparative ratings between Starbucks and other chains shows a narrower gap. In places where Starbucks has not competition however, it is likely to get disproportionately good ratings.

Let us say you are one of the contributors in the survey and are in St.Louis, MO. The competition for Starbucks in St.Louis is likely to be (I guess) the burnt robusta coffee at the local restaurant. In such a market, Starbucks will enjoy a clear advantage, both for the quality of the coffee as well as the ambience. So, let's say, you had to rate Starbucks on a scale of 1-5. It is likely you would give Starbucks a 4-5 in a non-competitive market, such as St.Louis, in the absence of valid benchmarks or competition to compare against. In a competitive market dominated by multiple brands, the difference between Starbucks and other brands is likely to be narrower. Also, the assertion can also be made that a more discerning audience (having had the opportunity to sample multiple chains) is less likely to give extremely high scores (4s and 5s) to any of the choices under consideration.

Therefore, the sampling design and the analysis methodology becomes extremely critical for surveys around this. To avoid the "no-competition" bias, there could a number of questions a market research analyst would need to ask herself:
1. Should we use only data points from places where there are multiple chains in the same geography? (Doesn't sound fair. We will be throwing away data, which a lot of sensible people have explained is a cardinal sin. We should probably weight the information in some way).
2. Should we consider data for the analysis only where a person has provided ratings about multiple chains voluntarily or penalize when people have not rated a chain that could have been rated?
3. Or are there modeling solutions available to manage this conundrum? Topic of my next post!


Anonymous said...

Never understood the point of these preference surveys. Isn't it possible to take the raw data from these surveys and manipulate it in any number of ways to spin an appropriate story?

Krish Swamy said...

Sure, but the agencies that produce these ratings (like Zagat, JD Powers) have built a reputation for providing quality, unbiased opinions. Zagat has a self-interest in making sure its coffee ratings are unbiased, if one is to pay attention to the Zagat ratings for restaurants in your city.

Which is why it is important to call out a few methodology points in the survey. How the conclusions were reached are sometimes as important as what the conclusions are? In this case, Starbucks finishing as the winner is no real surprise. But it is useful to know whether that is an all-US assessment or does it stand true for individual geographies as well? Is Starbucks #1 in most parts of the country or is it really #2 in most markets dominated by a popular local player?

Simon Morice said...

Is there really any such statistically significant entity as 'the best coffee'? If there is then doesn't the answer really segment into several different preferences? For example, irrespective of bean and roast, I don't like steam scalded coffee and others do.

It would be unscientific to say that the number of people who believe something has any impact on its 'truth'. Social proof is not scientific proof.

Anonymous said...

Appreciate the recommendation. Will try it out.

Feel free to surf to my web blog; Nutra Burn 10 Review

Anonymous said...

Incredible points. Great arguments. Keep up the great spirit.

Here is my web blog :: Buy bellagenix

Anonymous said...

Right away I am going away to do my breakfast, when having my breakfast coming over again to read other news.

Feel free to surf to my blog - Rvtl Eye Cream

Anonymous said...

My spouse and I stumbled over here different
page and thought I should check things out.
I like what I see so now i am following you.

Look forward to looking into your web page yet again.

Also visit my website - ORder Slim Green Coffee

Anonymous said...

I'll immediately grasp your rss feed as I can not in finding your email subscription hyperlink or newsletter service. Do you have any? Please permit me recognise so that I could subscribe. Thanks.

Here is my web-site: natural colo cleanse

Anonymous said...

I like what you guys tend to be up too. Such clever work and coverage!
Keep up the superb works guys I've included you guys to my personal blogroll.

My web-site :: Max Robust Exterme

Anonymous said...

hey there and thank you for your info – I have definitely picked up anything
new from right here. I did however expertise a few technical
points using this web site, since I experienced to reload the web site a lot of times previous to
I could get it to load properly. I had been wondering if your hosting is OK?
Not that I am complaining, but sluggish loading instances times will sometimes affect your placement in google and could damage
your high quality score if ads and marketing with Adwords.
Anyway I'm adding this RSS to my e-mail and can look out for a lot more of your respective fascinating content. Ensure that you update this again very soon.

Also visit my website - Beta Force Max

Anonymous said...

Hmm is anyone else having problems with the images on this blog loading?
I'm trying to find out if its a problem on my end or if it's the blog.
Any suggestions would be greatly appreciated.

Here is my homepage Profit web system review

Anonymous said...

This article will help the internet people for building up
new weblog or even a weblog from start to end.

Here is my weblog Buy nutra green coffee

Anonymous said...

This post is in fact a fastidious one it helps new net viewers, who are wishing for blogging.

my website ... best Wrinkle Cream

Anonymous said...

Hi to every body, it's my first pay a visit of this webpage; this weblog includes awesome and truly excellent material designed for visitors.

Feel free to visit my website; Slim XS Weight Loss Supplement