Peer evaluation is not objective: Academia and Law Firms

Academics believe that universities are meritocracies, or at least that they should be. And we’re not alone. Lawyers think the same about law firms. We all think that our workplaces should reward our accomplishments. If you’re the best researcher, you should win the competition for funding; if you’re the best lawyer, you should be promoted to partner.

The bad news, I’m sorry to report, is even worse than we might fear. Our systems aren’t meritocratic — they simply can’t be. On our own, we cannot avoid taking a rich set of factors into account, even when explicitly instructed not to.

As I learn more about careers in law, I am increasingly struck by the similarities to academia. And when it comes to peer evaluation, there is a growing body of research from both sectors showing that women and men are judged differently.

If this is true, it means there are structural impediments to career advancement for women. Women are not underrepresented among professors and law-firm partners because they don’t want those jobs; they are not underrepresented because they aren’t good enough. They are underrepresented because they meet stumbling blocks that their male colleagues don’t — and that’s enough to hold them back.

Two studies — one from each sector — lead us to this conclusion. Both provide empirical evidence that reviewers, evaluators, and peers require more of women than they do of men. They show that men and women who are evaluated equally in fact have quite different profiles, and that the women have to do much more than a male colleague to be viewed as equally meritorious.

Monica Biernat, M. J. Tocci and Joan C. Williams demonstrate in their new paper The Language of Performance Evaluations: Gender-Based Shifts in Content and Consistency of Judgment, in the journal Social Psychological and Personality Science, that the evaluation of junior attorneys is carried out differently for men and women. The evaluation process yields a numerical ranking for a candidate, along with a written evaluation. The numerical ratings determined by male supervisors were consistently better for men than women. But the prose descriptions showed either no difference between sexes, or a difference that favored the women. (The article is also summarized on The Careerist.)

In academia, one of the best demonstrations of the inevitability of bias is from the work of Christine Wennerås and Agnes Wold. They published a paper called Nepotism and Sexism in Peer Review, in the journal Nature. Wennerås and Wold studied the evaluation forms filled out by reviewers who were rating applications for post-doc funding through the Swedish Medical Research Council. They standardized the portfolios of the applicants and compared them based on the sex of the applicant. Their research demonstrates that women had to be 2.5 times as productive as men to be judged as equal to the men.

When I talk about quotas — as I did at the first European Gender Summit — I often ask audiences what their biggest objections to quotas are. One of the most common answers is that the introduction of quotas into systems makes those systems unfair. When you have quotas for sex, men and women are judged by different standards.

That claim is either exactly wrong or exactly right, but in either case, we can’t be satisfied with the situation we have.

The claim is exactly wrong if it implies that our system today judges everyone on the same criteria; research such as that discussed above makes it clear that men and women are not judged by the same criteria today.

But the claim might be exactly right, if it means that we already have quotas. Men and women, after all, are demonstrably judged by different criteria. If that is the hallmark of a system with quotas, then we’re already there.

But the quotas we have now are for men.