The real oversights in Lenny Rachitsky’s lists

Yesterday, I wrote about Lenny Rachitsky’s attempt to figure out which companies produce the best PMs and the problems with his analysis. But today is more important: even if you correct for method errors, this data doesn’t really answer the question he is asking. But it may tell PMs, particularly those underrepresented in tech, what companies to avoid.

A brief reminder of method: he looked at PMs who have left a company and what happens across 7 career categories, like how quickly they are promoted in their next job.

But using only alumni data introduces significant confounds. And he acknowledges this deep in one section of his analysis: “Another explanation is that the best PMs at FAANG companies are happy and don’t leave, and so we don’t see their trajectories in the data.”

This is very much burying the lead. All good insights start with removing as much systematic bias as you can from your sample and looking at only alumni means ignoring all the reasons that PMs choose to stay at a company. Many of the companies have experienced layoffs. Some are better at retaining leaders vs juniors. Companies that are newer have less time to show attrition.

And these biases aren’t random. Take Average Time to First Promotion. In his view, lower is better: it means the company turned you into a talented PM. But it could just as easily mean that a company systematically underpromotes top talent, causing them to leave and get quickly promoted elsewhere. This is particularly true for underrepresented people, who are most likely to be overlooked. 

But what if, by combining Time to Promotion and Leadership, we try to find companies that systematically underpromote talented people?

There are caveats. Smaller companies might not have as much room to promote people and we don’t have data in this sample to control for that. Cross-validating with another dataset (like time in role without promotion before leaving) and qualitative interviews would go a long way.

But let’s say we do believe the combined metric is reasonable. Where should folks choose to work?

Worst first; these companies appear to systematically overlook top talent.

46. Discord

45. Deel

44. Revolut

43. Scale AI

44. Plaid

Both Revolut and Plaid made Rachitsky’s Top 5 Best Companies. This is why looking at things through an inclusive lens is so important: otherwise you might not just give random advice but advice that is actively bad for disadvantaged groups.

The best companies?

1. Microsoft
2. Adobe
3. Apple
4. Intuit
5. eBay

Maybe large companies with more formal processes are better at mitigating promotion bias. Maybe they just have more room to grow. Maybe underrepresented people can more freely transfer internally away from biased managers there. I don’t feel strongly enough about this data to feel like I know the answer.

And that’s really the point: good causal analysis matters and being wrong can worsen systematic issues. So if you have a large audience, be particularly careful what you say, and don’t rely on others to check your work.