NPS - Taking a Meat Cleaver to Slicing Sushi
I prefer food analogies, because it makes for a more interesting story, and food is something everyone can relate to, whether friend or foe.
For over two decades I watched how NPS (Net Promotor Score) was revered and adopted across organizations and businesses worldwide. From middle managers to c-suite executives, who reported on these scores weekly, and sometimes even daily, as a testament to their understanding of the voice of their customers. Its ingenuity was supposed to be in its simplicity, because executives could not be bothered by white noise. Market research, no doubt, is most compelling when it’s simplified, but let me explain why simplicity in its rationale should never remove the purpose of its creation.
As someone who has been in the market research industry for over two decades, on the supplier and client-side of research, businesses often ask how is it that customer defection grew while NPS metrics stayed the same or was increasing. How could this be and why did NPS not reveal increasing customer defection and attrition? Let’s go back to sushi for the explanation.
In Japan, if you want to be a sushi chef, the road is long and arduous. You will spend several years just learning to make the rice… gentle brushings to wash the rice, blending the perfect vinegar seasoning, and delicately folding the seasoning into the rice at the perfect pace and temperature, while the rice is fanned with a delicate breeze. The proportion of rice, seaweed and wasabi is very exact, so the balance is perfect. Then you’ll spend years learning how to select high quality sushi at the fish market and choose the best knives. Thereafter, you’ll spend a lifetime to clean, slice and plate your creation. The learning actually never ends - you’ll be a student for life.
The underlying current of market research is very similar - you’ll spend decades learning the fundamentals, but a lifetime honing the art and the craft. There’s always a better way to ask a question, in person, online or on paper. There’s always a better way to interpret or craft the story around the insight. The art and science of market research is complex, from understanding the fundamentals of math and statistics, to designing the unbiased survey instruments, to preserving data integrity in collection methods and then weaving that into a compelling and impactful story.
Even before you craft a survey question, who you ask, where it goes, what you ask before and after, what instrument you use to ask it… all has consequences that impact the integrity of your responses. There are countless publications out there on the right scales to use and how to label them and space them. Yes, even the white space matters in surveys. With the wider use and popularity of market research, more companies invested in their own in-house market research teams and departments. Traditionally, third party suppliers that conducted customer loyalty programs offered some degrees of independence, non-bias design and data collection and integrity driven interpretations.
Overtime, as these instruments and methodologies were brought in-house, it was harder for researchers to maintain research integrity. Siloed organizations had more internal business conflicts, which gave way to more non-researchers designing and implementing research. The benefits were that companies recognized the need for market research and there was more collaboration at various phases of marketing, product innovation and service. But the drawback was that the quality of the research declined significantly, under the rationale of simplicity.
Like sushi restaurants, the more there were, the less consumers realized the art and science behind the product. Sushi could be consumed in mass quantities, like big mac and french fries at your local McDonald’s.
As more companies adopted market research, the science and craft behind its current day applications became mainstream and further simplified. No one could tell the difference between well-designed research and insights - any business department could throw together their own Survey Monkey for any and every question they needed a quick answer to.
It was no longer if we should make X, but rather, what color would you want X to be… red, white or blue? Market research was no longer consolidated into one main department, but every business that had budget felt they could run their own surveys and collect their own insights. Cheap data collection tools made surveys accessible to everyone.
Business owners wanted to ask … “How much do you really like our products and services”.. instead of “what do you think of product X? What do you think of the services Y?” Subtle differences that have drastically different results.
Running statistics was also made incredibly easy, just click a button to run a correlation in Excel or in SPSS, it didn’t matter where you got the numbers or how they were collected. Historically, to run the mathematical calculations behind a Factor Analysis, anthropologists and archaeologists would need a team of mathematicians and several months of data crunching and cross-validations. With the advent of super computers, several months of calculations by hand could be completed in several seconds. Consequently, there grew increasing competing customer voices throughout the organization as to who represented the true Voice of the Customer.
NPS came along at a optimal time - it was a simple solution to a painful problem, that could be wrapped neatly in a bow and marketed pervasively across the organization. It was published in the Harvard Business Review - so it got instant credibility and most folks could pitch it without understanding closely the mechanics underneath the hood of the moving vehicle.
Only one important question needed to be asked…. Is NPS tied to loyalty? If the answer was yes, the assumption is that raising NPS would raise loyalty, and hence your revenue and/or customer base. But the true answer is not so simple with NPS. And that ambiguity alone makes it a flawed metric for customer loyalty.
Let me first explain how you calculate NPS.
You ask a customer how likely they would recommend a brand, product or service, and have them rate it on a 0 to 10 point scale.
It seems simple enough, but let me explain two fundamental flaws with this loyalty metric.
Flaw #1: If there are multiple ways to calculate the same NPS score, and multiple ways to interpret the same NPS score, it CAN NOT be a reliable loyalty metric
With NPS, there are multiple ways to calculate the same NPS value, so how do you exactly interpret that score? Well, you don’t, not consistently anyway.
Scenario A: If I had 10% Detractors and 70% Promoters, my NPS is 70-10=60. So 1 in every 10 customers is a detractor - I seem to be doing very well in my business.
Scenario B: If I had 20% Detractors and 80% Promoters, then 80-20=60. Similar to Scenario A, it seems like a high NPS score. But now, 1 in every 5 customers is a detractor. If I was a restaurant, my 1 in every 5 customers would be sharing their restaurant horror stories to least 5+ other people, by mouth, on Yelp, on Facebook and Instagram. I’ll probably be out of business in a year.
Interestingly, if all of my detractor customers migrated from 0-3 scores to scores of 6s, my NPS value would have stayed exactly the same. I would not even be able to see slight improvements in my recommend metrics because NPS is like a butcher knife trying to slice sushi.
Is it still sushi if you hack it with a butcher knife?
Well, it depends on whether you get your sushi from 7-Eleven or from a Japanese trained sushi chef.
Flaw #2: A loyalty metric needs to be calculable at the individual level in order for it to be effectively linked to financials or customer growth.
Let’s assume I give this fundamental flaw #1 test a pass, and do as Reicheld had mentioned, just ask customers that one metric and then ask why they feel that way and I’ll be able to resolve all of my customers’ pain points.
In a given day, if I’m Amazon - I may receive millions of transactions worldwide, and an exorbitant amount of verbatim comment feedback to why they gave the score they did. Even if I was a smaller online retailer, I’d still have hundreds or thousands of transactions per month. I could have my teams read through all the verbatim comments relevant to their business unit and I’ll fix everything and make my customers happy. But, I only have a fixed budget to address all the customers’ problems, which ones should I prioritize? Should I pour 30% into fixing my Cart Checkout or to On-Time Delivery? Which one is going to increase customer retention and repurchase?
Well, if I ran a simple regression on my Likely to Recommend Metric - and some other metrics, I would have this prioritization quantified. But wait, I can not do this, because according to Reicheld, I only needed to collect one metric. And even if I did collect other metrics, I simply CANNOT run a correlation or a regression on NPS, because it’s a score that can only be calculated at a group or aggregate level e.g. by product category, or by month, etc.
There are no NPS values per customer, because it is a value created by a summation of aggregated parts. And because of this fact alone, I can never use statistics to calculate viable correlations, linear regressions, cluster analysis, or anything else remotely in that category.
Without individual customer level metrics - you can't really make good use of statistics.
So what? Phasing out NPS is not going to help my business solve real customer problems.
How am I supposed to unwind out of the NPS metric?
I will have no benchmarks in which to use for the last several years and what should I use going forward?
A simple, but effective solution is just just to use an average or mean score from all the Likelihood to Recommend metrics you’ve been collecting. You’ll have a score on an individual level, 0 to 10, and on an aggregate level, you'll have average scores that can be leveraged in numerous statistical applications to help you prioritize your budget. You can even use these simple mean scores to run correlations or regressions with financials to see how strongly raising one percentage point ends up raising revenue.
The science and viability of the Likelihood to Recommend metric as a loyalty indicator has been around for a long time. You can keep it simple but scientific - by using a mean score or scaling it to a 100. Track it, along with several other key metrics as identified by your customers and your business. Retain the simple science, even if you give up everything else.