There are majorly two questions you should ask of any metric:
1. Is it a valid metric?
A metric isn't valid if
- Some scenarios lead you to closer to the ideal outcome but do not make progress on the metric.
- Some scenarios lead you to hit your target metric but actually take you away from your ideal outcome.
- Hitting your goal metrics doesn't materially matter to your business.
2. Is it feasible, i.e., how easy is it to measure?
Take the example of Netflix.
One of their hypotheses was that a simpler member experience would improve retention, which was their high-level metric.
But how do you measure “simple?”
And how do you show that it improves retention?
The Netflix team began exploring customer service data.
- Why do members call or email Netflix with questions or complaints?
- What links do they click on when they visit the help pages?
- Where do customers get confused?
This was Netflix's plan of action:
- They talked to new members in one-on-one sessions and focus groups.
- They asked a small group of customers to write a journal describing their weekly activity with Netflix.
- They looked at existing data for the new member sign-up flow, as well as their first few weeks with the service.
After these exercises, the team realized that there was a point of confusion among new members: Netflix's early DVD-by-mail service required customers to create an ordered list of movies that we would send to them.
But some new members failed to add any videos to their Netflix “Queue.”
Some new members chose a plan, entered their credit card information, then asked, “Now what?”
The notion of adding at least three titles to their Queue confused many new members. The Netflix team identified that simplifying the sign-up process would make it easier for customers to create a list of movies.
The proxy metric they devised was
"The percentage of new members who add at least three titles to their queue during their first session."
After making the sign-up simpler, Netflix saw that 70% of new members added at least three titles to their queue during their first session. By the end of the year, after a series of fast-paced experiments, they were able to increase this percentage to 90%.
Over the same period, they drove month one retention from 88% to 90%. Both the higher-level metric (retention) and the "simple" proxy metric moved together.
Takeaway:
Directly using a high-level metric like retention for all projects may not always be feasible.
Lower-level metrics — proxy metrics — are easier and faster to move than a high-level engagement metric. Ideally, moving a valid proxy metric will improve the high-level metric (e.g., retention for Netflix), demonstrating a correlation between the two.
Later, you can prove causation via an A/B test.
