Jun 9, 2017 · 13 minutes

A good rule of thumb when you do research that shows a dramatically different outcome than previous studies is to ask yourself if there is something in your methodology that could have caused that.

That’s especially the case if it’s about something as culturally loaded and complex as gender or race.

Last week, Bloomberg published a study that purported to show that female VCs are not more likely to back female founders. This flies in the face of previous research that shows that firms with a single female investing partner were twice as likely to back a female founder.

At the time, I Tweeted to the author and editor about why I found the methodology strange and flawed, and mostly moved on. But the article has come up again and again with female investors I’ve spoken to since its publication, rolling their eyes at the perpetuation of the outdated idea that women don’t help women. Seriously, still in 2017?

I feel increasingly queasy with the idea that it’s out there ready to be Googled, and used as a defense for why hiring women in venture capital doesn’t matter after all. Used to further perpetrate the myth of the “Catty woman” and the “Queen bee.” You know, the bullshit justification used for so long that it’s really women who are keeping women down.

That rationale, too, was once based on “research.” Only it turns out it was similarly flawed. From a piece by Sheryl Sandberg and Adam Grant on this topic:

According to the queen bee theory, a female senior manager should have a more negative impact on the other women trying to climb into professional ranks. When strategy professors studied the top management of the Standard & Poor’s 1,500 companies over 20 years, they found something that seemed to support the notion. In their study, when one woman reached senior management, it was 51 percent less likely that a second woman would make it.

But the person blocking the second woman’s path wasn’t usually a queen bee; it was a male chief executive. When a woman was made chief executive, the opposite was true. In those companies, a woman had a better chance of joining senior management than when the chief executive was a man.

In business and in government, research supports the notion that women create opportunities for women. On corporate boards, despite having stronger qualifications than men, women are less likely to be mentored — unless there’s already a woman on the board. And when women join the board, there’s a better chance that other women will rise to top executive positions. We see a similar pattern in politics: In Latin America between 1999 and 2013, female presidents appointed 24 percent more female ministers to their cabinets than the average for their region.

The myth of the queen bee also failed to take into account what happens when any minority are put into a zero-sum situation, where there is only one seat at the table for an “other.” It failed to take into account the power of benevolent sexism-- a way of rewarding women who help prop up the existing system, not making too many waves about it needing to change. It’s about the psychological impact of exclusion and bias, not anything having to do with women, specifically.

Susan Wojcicki has argued the same, most recently in Vanity Fair:

 [T]here is a solution that has been proved to address gender discrimination in all its forms, both implicit and explicit: hiring more women. Employing more women at all levels of a company, from new hires to senior leaders, creates a virtuous cycle. Companies become more attuned to the needs of their female employees, improving workplace culture while lowering attrition. They escape a cycle of men mostly hiring men. And study after study has shown that greater diversity leads to better outcomes, more innovative solutions, less groupthink, better stock performance and G.D.P. growth.

Indeed, in China, where women have double the percentage of board seats and C-level jobs in tech companies, a higher percentage of companies have formal diversity programs to do even better.

We see the same phenomenon in politics: Iceland was number one on the Economist’s Glass Ceiling Index, which tallies up a basket of statistics showing how well women are treated at work. And yet, Iceland wasn’t content. It subsequently passed a law requiring companies to prove they are paying workers equally. Said Iceland’s (male) prime minister, “We may rank number one in the world at the moment, but the job is not done still.”

So there’s little evidence in any culture or industry I’ve studied that the advancement of more women in senior roles is bad for equality. Quite the opposite: The more power women gain, the more they tend to demand and push for even more equality.

So how did Bloomberg supposedly prove the inverse? Arbitrary and messy methodology.

I don’t actually think Bloomberg was trying to twist the data to make their case, but I could understand how someone could jump to that conclusion.

For one thing, they continually seemed to conflate whether they were trying to measure the investing patterns of female partners, versus firms that had at least one senior female investing partner. Picking one of those to measure and sticking with it matters, because data has shown that firms with at least one female investing partner see more deals headed by female founders.

If a woman pitches a firm because it is inclusive, and a male partner happens to have the more relevant industry expertise, that doesn’t mean the role of the female partner at that firm was irrelevant. Because deals are generally debated and voted on by an entire partnership, the role of women in the room when decisions are being made matters greatly. For one thing, it means female founders don’t have to suffer the ubiquitous and humiliating act of the executive assistant getting pulled into the room to ask if she’d use that service. A female partner could be consulted instead.

Mike Maples of Floodgate says his firm gets pitched by far more female entrepreneurs than most firms, which he argues is because Floodgate is a gender balanced firm. About 20% of their pitches come from female founders, a stat that he says his friends at other firms simply don’t believe. They tell Maples they see 1% or 2% of their pitches from women. So Maples-- a man-- is more likely to back female founders, simply because he has female partners.

Most VCs exist in a partnership. They make decisions as a partnership. Just because it was a man in a firm who did a deal doesn’t mean the deal would have been done if the firm didn’t have female partners.

Bloomberg also picked an arbitrarily small sample set of 17 “top” firms. What’s so wrong with that? A lot of things, but most notably, its limited sample set astoundingly left out five of the top six female investors in the world, according to the 2017 Midas List.

In other words, Bloomberg did a study about how “top” female VCs back female founders, leaving out almost all of the most successful female VCs in the industry. It didn’t occur to Bloomberg to measure “top partners” in terms of the female investors’ actual track records, but rather whether they belonged to the biggest, most historically dominant firms. Which is a shame, because those are the firms that are the most entrenched in how things are done in the past.

That in and of itself is a biased way of valuing a female VC’s stature in the industry: She’s considered a “top” partner only if she’s surrounded by the right men?

What’s more, the firms they did measure were wildly different: Some were high volume firms where partners who make a deal per month, others are firms where partners may only do one to two deals per year. The ability to say yes 12 times in a year, versus once or twice dramatically changes how likely a woman -- or any minority-- is to get that shot. Particularly given the fondness for serial entrepreneurs in Silicon Valley. If a firm does six times the investing volume, and hence funds more women, you could argue they are just funding more of everything.

Here’s the weirdest part: Bloomberg tried to correct for the difference between the firms it measured, but it did so in a bizarre way. Rather than looking at deals made per firm or even per partner it looked at the number of partners in a firm.

As a result, they greatly skewed the data.

Their first chart-- with the raw data showing how often these top firms backed women-- showed that indeed, four of the top five firms who backed female investing partners had at least one senior female partner. But when they readjusted it to take into account how many partners were in a firm, firms like NEA fell out completely, because it’s a mega firm. Under the new methodology, the top firms in backing women were Felicis, Lowercase, Index Ventures, and First Round Capital, none of which have a senior female investing partner.

This was Bloomberg’s “evidence.” The catty smoking gun.

Case made! Meeeee-oww!

Only, that conclusion makes little sense if you understand how these firms work. For one thing, First Round does extremely high volume, so comparing its decision making to a firm where partners may back one or two things a year is…. bizarre. Also First Round and Index are two of the more active firms when it comes to ecommerce investing, which is comparatively female-dominated. So there was no consideration given to sector-bias. And, Lowercase is hardly a typical firm, having made the bulk of its returns through buying and selling pre-Twitter IPO stock. (Also, it’s now defunct.)

And that’s not even the end to the weirdness of Bloomberg’s conclusion: Until recently, Felicis did have a female investing partner, and this survey relies on past data. So that’s moot.

And First Round has recently hired Hayley Barna, co-founder of Birchbox. The article notes that she’s a venture partner, a tier under senior partner, and hence didn’t measure up the arbitrary line the study drew. But it also noted that she has lead four deals, two in female-founded companies.

Again, it’s strange methodology: If she is funding women, why isn’t that relevant to the study? Why would her title in an industry where titles vary widely by firm matter more?

Most studies measure female “investing” partners, not female senior partners, because many women are just now breaking into the industry. There’s no good reason why Bloomberg would have picked only senior partners. And it greatly skews the data since this is a male dominated industry where men have the most seniority. All that should matter in a study about whether women back women is whether you have the capacity to write a check at your firm. I guarantee for the founder, actual money in the bank matters more than the title of the person writing the check.

To sum up: If Bloomberg had tried to correct the data by deal volume, not number of partners, or had taken into account the fact that Felicis had a female partner over at least part of the period it was studying, or had measured “investing partners” not senior partners, or had counted only still active venture firms, none of their conclusions would have held up.

I am truly not trying to beat up on Bloomberg here, but I’m sick of “research reports” that slice and dice the data in their own creative ways when it comes to gender, releasing results that could be used to rebut the case that female advancement matters in tech and the venture world.

This topic matters greatly. The venture industry is under pressure to have more diversity and invest in more women and minorities. Research that shows making their own ranks more diverse will lead to more diversity in their portfolios gives VCs a specific action they can take to achieve both ends: Hire more female partners and partners who are people of color. This has become a crystal clear rallying point for anyone who wants change. An unequivocally good thing for the industry, its returns, and fundamental fairness in the world.

Why twist data until you could prove the opposite?

Bloomberg did at least try to get out some of the cultural causes of why women sometimes feel pressure not to back women.

They rightly quote women who say that it’s far harder to pitch women than men. In my direct experience and that of most founders I know, it is quicker and easier to get a check out of man than a woman— no matter what gender you are.

And there’s good reason for that. Within large firms, female partners typically aren’t the rainmakers or most senior members of the team. At smaller boutique, women-run shops, women still have the burden of having a lot to prove. They tend to be more analytical and less gut-centric than men. Their experience is more likely to come from investment banking than being founders themselves.

That isn’t necessarily a bad thing given that whole gut-feeling thing also leads to a shit load of unconscious bias. But it means the process moves more slowly, certainly in my experience as a CEO. The flip side of that, according to many entrepreneurs I know, is that the same VC who makes a more educated thoughtful decision to invest is frequently a more thoughtful educated board member.

The idea that people who represent such a small minority of an industry wouldn’t feel more pressure to make more thoughtful decisions is… absurd.

There is likely a direct correlation with female VCs having to or choosing to be more deliberate in their decision making and female VCs in general out performing men, despite they frequently get pigeon-holed into areas that traditionally have lower returns like biotech and ecommerce.

There’s another force at play here too. Female VCs frequently feel pressure from their fellow partners, other VCs, and even in some cases their limited partners to prove they do more than just “female deals.”

Every single female investor I know has complained about the sheer volume of Tampon deals they’ve gotten in the last year. It’s a catch-22: They love deal referrals, and they never want to discourage them. But they want deals that aren’t only “female” deals. They want to back and support women and invest in things they’d be customers of. But they want to be able to do more than that. That’s remarkably hard in an industry that over-relies on pattern recognition.

Let’s assume Bloomberg’s data was even directionally right and not skewed by strange methodology choices. It’s still an unfair lens, given the very real factors female VCs face if they back “too many” women and the massive disparity in gender in the industry.