First published in Oct 2018 on LI

I once interviewed at LinkedIn. I wasn’t very interested in the role; the recruiter approached me, and I had been out of the interview market for some time, so I decided to go for a refresh of my interviewing skills.

Let’s just say the process didn’t go well. I ended up having a “disagreement” with one interviewer about A/B testing—specifically, what p-value exactly means. His answer was that the p-value is alpha (Type I error), which it clearly isn’t.

I responded by stating that the p-value is a conditional probability and tells nothing about whether the hypothesis being tested is true.

The interviewer disagreed, and I told him his understanding was incorrect. Later in the afternoon, after I had left, he sent me a LinkedIn message admitting he was wrong. I appreciated that quality in him.

In the next round, during the first five minutes, I started on a slightly wrong footing with the Head of Data Science. I shared my prior interview discussion, and his reaction suggested that he didn’t find it credible that one of his data scientists could be wrong. Maybe he didn’t like an interviewee calling out an interviewer.

He later asked me a case-based question: What should a data scientist suggest to a product manager when the data-backed solution could cause millions in business losses?

I answered: I would still suggest going ahead with implementing the feature, even if it meant losing money, because my sense says it’s going to be a good user experience in the long term and will build trust in users.

He didn’t like my answer. He kind of mocked the idea, saying that as a data scientist, I was suggesting my sense instead of data-driven insights.

I said that’s fine, but I would still personally recommend launching the feature.

There is a belief in building products that you should ALWAYS rely on data driven hypothesis. That's largely true but that’s not how great products features are built all the time.

Early in Netflix’s product life, Netflix introduced a one-month free trial for new users. The homepage was designed to calm potential subscribers, reassuring them that they could cancel anytime. However, to access the trial, users had to enter their credit card details, and they would be charged once the trial ended.

At that time, Netflix’s retention rate after the first free month was 90%—a huge success. However, there was no email or text notification system** to remind users that their card would be charged when the trial ended. This led to many customer complaints and call center requests, costing Netflix around $10 million in handling these issues.

To address this, Netflix experimented with a user notification system, sending an email a few days before the trial ended to remind users of the upcoming charge.

What do you think happened?

The retention rate fell from 90% to 85%, resulting in a $60 million annual loss for Netflix. Even after accounting for call center cost reductions, the net loss of this feature was around $50 million annually.

So, what did Netflix’s Head of Product do?

Contrary to conventional wisdom— “always build products based on data-driven testing” —Netflix’s product leader implemented the email notification system anyway**.

Obviously at that time many would have felt that this guy is smoking something, who is going against what the data clearly screams out - retention rate falling if you implement this feature and would cause losses in millions to company.

But as a product owner, he took a call to implement the feature. I would have done the same and it was a similar thing what I suggested to that guy at Linkedin.

Great product features are not those which are always built on data driven insights. Great products features are built for User Trust.

Your users will keep using your product if they trust you, believe in you, and know you act in their best interest.

In India, Flipkart was the pioneer in e-commerce long before Amazon entered the market. However, within three years of Amazon’s arrival, Flipkart started losing market share.

Flipkart’s CEOs and founders blamed Amazon’s massive cash backing and even requested the Indian government to create barriers for foreign companies.

But the truth is, Flipkart didn’t lose because of funding issues. Flipkart lost because it eroded user trust.

For a failed delivery, Flipkart customers had to go through a hassle before getting a refund. Over time, this eroded their trust, and the goodwill Flipkart had built started to fade.

When building a product, the number one priority should be optimizing for user trust. This requires a long-term focus.

At times, this may contradict short-term data-driven metrics. If you only focus on short-term metrics, you may optimize for short-term gains at the cost of long-term success.

Nothing makes a product more successful than a satisfied and happy user who trusts your product and service.

Building user trust is not a short-term metric. It is built patiently, with a long-term focus.

And sometimes, optimizing for user trust means going against what the data suggests.

You can only make these contradictory decisions where data suggests one thing but user trust suggests another —if you build products with user empathy.

Good products are data-driven. Great products are data-driven but never at the cost of user empathy.

User trust is what makes products survive and thrive in the long run.

Thoughts on WeWork →