When Amazon started expanding from books into other categories, they initially assumed that a more extensive range of selections, i.e., offering more items on Amazon would drive more sales.
For the teams in charge of increasing the product range on the platform, this meant listing as many products and creating as many detail pages for them on the platform. Each detail page would have the product description, images, reviews, estimated shipping time... basically an older version of the product page you see on Amazon today.
So, the initial assumption of a larger product range = more sales translated into measuring how many new detail pages were being created. This was the metric Amazon was tracking. "More pages = better selection" was the thought.
Once the team identified this metric, the retail teams became excessively focused on adding new detail pages — each team added tens, hundreds, even thousands of items to their categories that had not previously been available on Amazon. For some items, the teams had to establish relationships with new manufacturers and would often buy inventory that had to be housed in the fulfillment centers.
But what the teams soon realized is that an increase in the number of detail pages, while seeming to improve selection, did not produce a rise in sales.
What had happened was, retail teams, in the pursuit of adding more products to the platform, had ended up adding products that weren't in high demand.
So, basically a Goodhart's Law type of situation.
In fact, instead of increasing sales, the cost of holding inventory went up as low-demand items started taking up valuable space in fulfillment centers that should have been reserved for items that were in high demand!
Amazon realized that it had chosen an input metric that didn't directly influence the output metric. They corrected this by switching the lens from how many products Amazon was adding to how much demand the platform was generating.
So, over time, the input metric changed from
- number of detail pages
- to number of detail page views
- to percentage of detail page views for products that were in stock
- to the percentage of detail page views where products were in stock and immediately ready for two-day shipping
Notice how through this process, Amazon went on refining its input metric and arriving at one that finally had a high correlation with the output metric they were trying to move the needle on — sales.
In general, the process of arriving at input metrics that matter and drive business goals is a process of trial and error.
You have a funnel that is wide at the top — with multiple marketing channels at the top of the funnel that have their own metrics — and narrows down as the audience turns into qualified leads, which then turn into high-intent prospects which then turn into paying customers.
At every stage of the funnel, there are multiple inputs. At the top, you may have Facebook, Instagram, and LinkedIn as your marketing channels.
Towards the middle, you have your website where customers can drop their lead, at which point you may run different initiatives to strengthen their intent. These may be marketing emails, sales calls, and running free events.
At the bottom, you may have a dedicated team of closers.
Of course, channels and tactics at every stage of the funnel may vary according to the type of business and product. But the point is that at every stage, there are multiple inputs going into creating the output, which is: pushing the consumer down the sales funnel.
Now, the way you figure out which input metrics are really driving the output metric is by defining the individual metrics and then measuring if growth in those metrics correlates to overall growth at the bottom of the funnel.
For example, let's say your hypothesis is, "Increasing Instagram followers will lead to more qualified leads."
But over a period of 3 months, if you find out that increasing your Instagram followers (input metric) is not creating more qualified leads on the website that are attributable to Instagram (output metric), you know that growth in Instagram followers is currently not a good proxy for growth in sales.
At the same time, you can also think about why that is the case: is your content attracting the wrong audience? Is your content building engagement but not intent?
If after trying various strategies on the channel, you still do not see more Instagram followers translating to more leads, you can confidently say that Instagram may not be a good channel for your product.
Controllable input metrics are only important if they drive desirable outcomes.
If the metric is wrong, or the metric stops driving output at some point in the future, it is best to drop that metric and the activity producing that metric.
In any case, the only way to avoid the kind of trap Amazon fell into above is to not dictate a metric to optimize in a top-down fashion without having first studied the process that creates the outcomes.
If you keep focusing on the input metrics without considering how it interacts with the larger process, you may have designers de-emphasizing the customer service button on the website, just because they were forced to somehow reduce the no. of service calls the customer care team was entertaining per week.
To actually do it right would require going into why the customer service team is having to handle so many requests in the first place. To do it wrong would be to simply keep watching the target and yell at your team for underperforming.
