For a tweet to go truly viral, it should be slightly wrong. Or it should lack enough nuance to let arguments and differing opinions flood in.
And Kunal Shah (Founder – CRED) seems to have done it with this tweet in particular:
But before understanding the meaning behind his tweet, let’s first try to understand what A/B testing means.
Imagine you run a business that sells products via a website. One day, your teammate comes to you and says,
"Visitors on our website are growing rapidly. But we are facing a high number of drop-offs on the homepage, so no one's moving onto the payment page. What should we do?"
On further enquiry, your teammate tells you,
"We are currently asking the user for their mobile number, but others are suggesting that users are typically not comfortable giving out their phone number so maybe we should ask them for their email address instead?"
Now, if you as a product manager see no clear reason from a product POV to pick one over the other (this is important and we'll come back to it!) and just want to maximize user conversion from the homepage to the payment page, the best option here would be to test both options using an A/B test.
A/B testing is essentially comparing two versions of a web page by splitting users to each version to figure out the better performing variation. So, in this case, half of the users selected at random will be asked to put in their phone numbers and the other half will be asked to put in their email address.
The goal is to see which version wins, i.e., improves your conversation rate.
A/B testing typically involves looking at existing usage data, setting goals to improve conversion rate, generating clear hypotheses behind both options, and running the experiment.
And you run the test for several weeks to account for random volatility in user behaviour or changes due to external events like festivals.
Now that you understand what A/B testing is and when it is used, let's come back to Kunal's point.
Remember I said that you run the test when you have no clear reason to pick one option over another?
Now, imagine the kind of product decisions you can make with this A/B testing approach. I bet only very small ones, like seeing which line of copy converts better, or which button placement works better.
But you cannot decide product- or category-defining features with A/B testing. That needs product vision and a very opinionated approach towards building.
Steve Jobs didn't come up with the idea of an iPod or an iPhone via A/B testing.
Ford didn't make the move from the horse carriage to the car by asking his customers and doing A/B testing. Elon Musk didn't come up with the idea of reusable rockets via A/B testing.
Moral of the story
You can only use A/B testing to optimize small product decisions. You can't use it to decide what the product itself is going to be. A product built that way will either be very conventional and boring, or it will simply fail in the market.
Of course, there are some industries like pharmaceuticals where trials akin to A/B testing are the whole game. But that is not the point he was trying to make here.
