Watson & Crick’s 1953 article in Nature revealing the double-helix structure of DNA was not peer reviewed. Many scientists claim this paper presents the most important discovery of the 20th century. The peer review system is what gives science integrity. Yet this paper was published based on the evaluation of the editors that it was obviously true.
One great discussion at the European Gender Summit in Brussels gathered editorial representatives from some of the most important scientific journals in the world, including the Lancet and Nature. That’s where I heard this story.
A pervasive theme at the Gender Summit was the connection between transparency and fairness, or between opacity and discrimination. When processes are not clear, open and transparent, prejudice comes into play, and the result can be unjust.
Where do these two topics overlap? Are there senses in which scientific publishing is secretive or opaque?
I think there are at least two areas where the guidelines are so vague that the (perhaps subconscious) biases of editors are likely to play a role.
The first of these involves journals that have professional staff members who carry out an initial sorting of submissions. Both of those journals report that they send out approximately 20% of submissions for peer review. How do they make that decision? They have scientific editors who work full time at the journal and these editors look at submissions and judge them on a number of criteria.
At the workshop I attended, a repeatedly used phrase was “impact.” That is, they identify articles likely to have high impact and send them out for review. Articles that are likely to have low impact, are returned. This is an implicit and nearly opaque process, and I wonder how it could be opened up.
It’s in the interest of the journals to only receive submissions that they find potentially interesting. How can one communicate the notion of “impact” or, even more difficult, the notion of “likely impact” to potential publishing scientists?
The second murky area I see involves the selection of reviewers. When an article is sent out, it is usually reviewed by three peers, and the opinion of those peers will be extremely important in the final determination of whether an article gets published. All editors are concerned to avoid conflicts of interest when they select reviewers.
But once we get beyond that, how does this work? Of course we try to identify “suitable” experts. But it is likely that the personal networks of the editors are extremely relevant. Given the power the reviewers have, perhaps a step in the right direction would be to remove the cover of anonymity.
Scientific publication has a hallowed status. At times, I think that is deserved. But at times, it’s just another old boys network. When it starts to look like that, we have to start thinking about new models. We’ll keep discussing this at the upcoming Munin conference. Maybe you’ll join us?
Share
7 Comments
Republish
I encourage you to republish this article online and in print, under the following conditions.
- You have to credit the author.
- If you’re republishing online, you must use our page view counter and link to its appearance here (included in the bottom of the HTML code), and include links from the story. In short, this means you should grab the html code below the post and use all of it.
- Unless otherwise noted, all my pieces here have a Creative Commons Attribution licence -- CC BY 4.0 -- and you must follow the (extremely minimal) conditions of that license.
- Keeping all this in mind, please take this work and spread it wherever it suits you to do so!
Interesting. As someone who is writing up a paper at the moment, this is almost discouraging, but, I’ll make it. 😉
It is discouraging. But one can choose which journal one submits to with care. Publishing, and the process around it, is important. But throwing oneself into the realm of opacity, is not. We have to both work to change the system, and work to survive in it as it is today. Carry on!
For all the reasons you say above, this is why many believe the days of lots of individual journals is numbered (and maybe even the days of journals are numbered). Trying to judge potential impact at the time of submission simply doesn’t work, which is why PLoS have decoupled the ‘seems ok’ from the ‘impact’ part of the review, with the latter being post-publication. This is further highlighted by the fact that 86% of what the Faculty of 1000’s 10,000 experts identify as the mlost important papers across biology and medicine are published in the journals that are NOT what most people would consider as the top-tier journals [I should probably state here that I work for F1000].
The ultimate approach to help improve the peer review situation is to make all peer review post-publication and open (with both a formal peer review and a community peer review) as then you have a continuum of views on how solid the work is and also on its impact (which of course could only become really apparent when other discoveries are made first), rather han an arbitrary cut-off date when a decision one way or the other is made.
These are very interesting thoughts. Just having finished a stint as editor for a top-tier journal in my field (Linguistic Inquiry), I struggle with the idea of post-publication reviewing. Or, rather, the absence of pre-publication reviewing. I think almost every article was improved in its readability and argumentation by the review process. But maybe that is covered by your “seems ok” criterion. How does that actually work? What is the process for reaching that conclusion? I need to learn more about F1000. Pointers?
So F1000 is post-publication peer review but of largely already pre-publication peer reviewed articles published in standard journals. You can view the Faculty tab at the top of, or About link in the footer of http://f1000.com.
We are however working on a new container-type OA ‘journal’ where all peer review will be post-publication and open. This crucially removes the long delays you often get between submission to your 1st journal of choice, and subsequent publication in a lower-tier journal. The key thing in response to your concern is that the authors can (and should) still improve their manuscript based on the formal open peer review as well as any comments from the broader community, and all versions, comments, author responses will be stored. In the meantime though, other researchers can already benefit from the manuscript being available. The ‘seems ok’ is about the type of questions being asked of the reviewer I.e. Is this scientifically sound, rather than is this interesting or an important development.
People are already increasingly posting their papers for comment from the community before formal journal submission – on Arxiv, Nature Precedings, researchers’ own blogs – so this is just a more formal extension of that where the whole discussion is in one place..
I think this is an important development, and it certainly is a way of extending not only the already existing practice you note, but the very spirit of the Web. One practical thing I wonder about, though, is whether the notion of “rejection” persists. Do article ever get (i) posted, (ii) post-publication reviewed, and then (iii) rejected? How does that work? Does the original submission remain. I’m the Chair of the Norwegian CRIS system (cristin.no) and I wonder at what point that system should acknowledge articles as published in a peer-reviewed forum. University budgets are tied to quantity of such publications. Thoughts?
I have talked to many people about what to do with articles that don’t get positive reviews and the overwhleming response has been that they should remain but make sure it is clear that the reviewers did not think the submission was of good quality.
As you say, the ability to state that something is now ‘peer reviewed’ is necessary, including for indexing in places like PubMed. I am therefore increasingly thinking that you need an ‘Editor’-type person to look at the reviews that come in and mark any that appear to have received a suitable positive response from the community as ‘approved’, and until that point, they remain ‘unapproved’. This also maintains an incentive to authors to improve their manuscript to reach the approved level, but keeps the whole process out in the open.