Oversights in Lenny Rachitsky’s lists

Newsletter expert Lenny Rachitsky tried to figure out which companies produce the best product managers, using a very large dataset. And not only did he arrive at the wrong conclusion, he inadvertently produced a list of places you probably shouldn’t work, particularly if you’re underrepresented in tech.

Preamble: I don’t know Rachitsky but our mutuals paint him as reasonable. And he had a chance to comment on this draft and gave it his blessing. The point isn’t to drag him – it is to understand how and why we have to do better when creating insights.

So today I’ll explain how his analysis actually miscategorizes the top companies. Tomorrow I’ll talk about why that doesn’t matter, because the data isn’t suited to answer his question anyway. But it can potentially answer another important question.

Let’s start with his stated intention: to guide people to great companies, where PMs learn the craft and go on to have stellar careers. To do that, he looked at PMs who have left a company and what happens across 7 categories: Total promotions, Fastest immediate promotion, Fastest rise to leadership, Highest rate of CPOs, HOPs, First PM hires, and Founders.

He then ranks each category and looks at the Top Ten in each. His overall Top 5?

1. Revolut

2. N26

3. eBay 

4. Plaid

5. Intercom 

Using rankings is his first mistake. If the difference between 1st and 2nd is small but 2nd and 3rd is huge, using ordinals obscures that. So you need to normalize to standard deviations with Z-scores, looking at how far companies are from the average.

Which leads to his second mistake: differences in population size. Deel has 79 former PMs on LinkedIn; Microsoft has 32k. Given the rarity of events like becoming a founder, this creates massive data anomalies if even one more person takes that path at Deel.

Finally, he assumes independence among his criteria. But this isn’t true. Take PMs from Discord. They’re almost dead last in terms of becoming founders but top the list in fastest promotions. Why? Because you can’t get promoted if you become a founder!

In fact, all of the data is substantially correlated. Average Time To Promotion and Average Time To Leadership have a correlation of 0.96; they might as well be the same list.

There seem to be two discriminant criteria: Rate of Founder/Head of Product/CPO/First Hire (Early Stage) and Average Time to Promotions/To Reach Leadership/Total Promotions (Late Stage).

So if we adjust to these criteria and use Z-scores, who actually comes out on top?

1. Revolut
2. N26
3. Intercom
4. Plaid
5. Palantir

This isn’t entirely different from Rachitsky’s list but it isn’t the same either. And with 230k followers and over $2m a year in newsletter revenue, what he says matters to a lot of people. So it is worth saying the right thing.

But it gets worse: even if you correct for bad statistics, this data doesn’t actually answer the question he is asking and may actually be an anti-signal that disproportionately affects underrepresented PMs. Tomorrow, I’ll talk about why.