I love talking to people as a way of designing products. At Thrive, I used to put a little note at the bottom of every weekly email that invited users in the New York City area to let me buy them lunch in return for just sitting down and talking about the day-to-day of how they related to their finances. With Avi Karnani, I’d chat up people in bars and collect stories about the best and worst of their money moments. Some of the most successful features we ever created came from just listening and watching.

In general, the tech industry has been moving in the same direction, trying to be more user-centric and responsive to their needs. But somewhere along the line, many companies made a critical mistake: they started listening only to their existing users. An emphasis on social media has made it the era of the “fan”, with products chasing the needs of the most passionate users.

The reason I call that a mistake is that I think it trades understanding for mere description. That is, focusing on fans tends to result in a list of directives, of favorite features and requests rather than insights into the process. Much like bad data science, it describes rather than explores, and it means features that ignore the needs that unite non-vocal users/non-users with super users.

As an example, let me use some real data from Do.com (special thanks to Jason Shah for agreeing to let me share it).

Do.com helps people run productive meetings, by setting agendas, documenting action items, and auto-summarizing. And prior to March, you had to either create an account directly or use Google to sign up. Their conversion rates were good, they were growing, and all was right with the world.

Like many companies listening to their existing customers, Do.com used UserVoice to collect user feedback and vote for new features, and while a few people had asked for Office 365 integration, it hadn’t risen to the top of the stack. Which isn’t surprising: if Google is your primary login, you’re going to end up with a big crowd of Google users, simply via homophile.

This is where Jason bucked the trend. Instead of simply relying on his existing users, he started talking to people who weren’t using Do. And a clear theme emerged: productivity-oriented users overwhelmingly used Office 365. So he got in touch with me, I set him up with some Office 365 API engineers, and Do ended up integrating Office 365 into their platform.

Two things happened. First, their conversion rates from the signup page went up significantly (7%). Because the only options had been creating an account or using Google, Do.com had been unknowingly turning away many new customers.   And because those customers abandoned pre-signup, their voice was almost never heard.

7% is a huge number, especially as far down the funnel as signup. If you imagine the amount of marketing money you would need to spend to drive a corresponding change by running people over the top of the funnel, considering what barriers you are putting in the way of signup seems dramatically more efficient.

The second thing that happened was that 25% of their existing userbase connected O365 accounts. Previously, because the only option was to hook up to Google calendar, all of their active users connected to Google. Once Office 365 was an option, Do saw an immediate reaction from their community and a corresponding rise in utility.

This is particularly important because it illustrates the limitations of passive user listening. If you only rely on users motivated enough to vote on features, etc. you ignore the fact that there may be a large group of people who want a feature but simply are too busy to tell you about it. Passive data collection relies on fans and fans have a myopic few of features. And in a world where there are an increasing number of options available for common tasks, you can’t simply live off early adopters and the passionate – you must also reach those who are seeking the path of least resistance.

I don’t want this to come across as decrying user research; I’m all for listening to users. And non-users. It is important that, as those concerned with creating behavior change in the world, we are not just passive listeners but active participants in the world that we want to change. It isn’t just good product, its good business – I’ll take 7% and 25% any day of the week.

Side note: Forget MOOCs; if you want to rapidly increase your knowledge about a topic, take a grad student to lunch. Most grad students are living off $25K or less and are eager for a good free meal – if you make it work on their time, you can find out virtually anything in 1:1 format, from someone who has more passion about the topic than the average online facilitator. Plus, you might just make a new friend.