Reviewed to Reviewing: Responsibilities, Considerations, and Quandaries of being a Reviewer for Academic Publications

This post is the second in a series from guest contributors Erin Renee Wahl and Dr. Rene O. Guillaume. See their first post here.

We are back again to talk about what it is like to sit on the reviewer side of an academic journal! In this post, we peel back the curtain on common problems reviewers face, from unrealistic timelines to lack of compensation. Most importantly, we consider how reviewers can embrace their role as partners to authors rather than gatekeepers. The post tackles issues like assessing your scope of expertise, aligning with journal values, and advocating for practices that support reviewers. Much like the author side, your experience as a reviewer will vary from journal to journal. Some will be more relaxed, and some will have strict rubrics. Some offer a ton of support and others are hands-off. But even on the reviewer side you still have a lot of choices about how you engage with the process. 

The peer review process is a critical part of the publication process for academic manuscripts. It is a system by which experts in the field assess the quality, validity, and significance of a manuscript before it is published. The peer review process typically involves the following steps:

1. The manuscript is submitted to a journal.
2. The editor of the journal reviews the abstract of the manuscript to determine its relevance to the journal’s scope and readership.
3. The editor selects two or three reviewers who are experts in the field of the manuscript.
4. The reviewers are sent the manuscript and are asked to provide a confidential review.
5. The reviewers’ comments are sent to the author(s) of the manuscript.
6. The author(s) revise the manuscript in response to the reviewers’ comments.
7. The revised manuscript is sent back to the reviewers for a second review.

The peer review process is a valuable tool for ensuring the quality of published research, but it is not without its shortcomings, including bias, lack of expertise, and inappropriate pressure.

In this next section, we’d really like to tackle three main things: considerations of peer reviewing, common problems, and how we can take ownership over our peer reviewing responsibilities and work to make the whole process better. Our goal is to spark reflection on how peer reviewing can embody core academic values of rigor, discourse, and mentorship. By being thoughtful about their responsibilities, reviewers play a key part in constructively shaping scholarship. We (Erin and Rene) talk about this all the time! And so we recorded one of our conversations and transcribed it below for the Librarian Parlor readers. 

Erin: So when do you decide to make the shift from author to reviewer? How would you judge that? For me, I love reviewing, and I love supporting writers. But I also feel more confident in doing that, and confident in knowing what I know and what I don’t know. I’m not gonna accept an article to review if I don’t know enough about this topic to judge whether it is good or not.

Rene: Yeah, and I think that’s one of the problems we run into with the peer review process as a whole. If you look at different online communities that focus on publishing and the publishing process, one of the things you see more and more are individuals who are not already connected to a journal being asked to peer review. I think a lot of folks are simply at capacity, and what happens then is you have individuals that are being invited to serve as peer reviewers in areas that aren’t always or aren’t necessarily in their area of expertise.

Right now I’m on editorial boards for two journals. I’ve gotten 5 invitations in the past 4 months to serve as a peer reviewer and only one of those has come by way of one of the boards that I’m on. Sometimes, when you submit your manuscript it’ll ask if there’s anyone in the field that you would recommend as a peer reviewer, and I don’t know if individuals are choosing folks, or if editors are just really kind of at their wit’s end and not finding enough peer reviewers or getting denied from their own board so that they’re really having to cast a wide net. I remember getting an invitation my first year on the tenure track, and I just didn’t feel prepared. It was in an area that I knew well and so I could speak to the paper. But I just didn’t feel like I had the experience. I had the expertise, but not the experience. But I found, as I went into year 2 of the tenure track, year 3, year 4…those invitations don’t slow up, and it’s not always because I’m recognized as an expert in the field. It’s because editors are having a hard time finding individuals to participate in the peer review process.

Erin: And if you look at journal websites, too, there’s very little on most about how they pick their peer reviewers, how they judge criteria. I know I’m typically asked to self-identify where I have expertise rather than somebody choosing me for that particular reason. You may have less expertise than you think you do if you self identify in some cases and that’s a problem. I wish we had a little bit more clarity on how peer reviewers were chosen, and what the process was like from more journals. Also the process which we laid out briefly in this article already isn’t always communicated. But there are also some editors who are way more involved. Some just send the articles out and say, you have 30 days to respond. Some actually say: “we want you to review this article, say yes or no,” and then the clock starts, and if you say no, then they go on to somebody else. This approach is respectful of the author, because then you’re not waiting around as long for your reviews to come back in. Maybe that’s something to consider when we consider if we’re ready to move into a peer reviewer spot. What is the journal itself like? Do they support their peer reviewers, too?

Rene: Yes! Or are the editors available for questions? I agree because I would say of the random invitations that I’ve gotten in the past couple of months, in many cases I’ve not even heard of that journal. I wasn’t aware of the journal, didn’t know any of the editors, and I think that’s one of the challenges too. You know, there’s simply hundreds upon hundreds of journals, which you know has some pros to it, but I think some definite cons. And one of them is not enough peer reviewers.

Erin: Yeah. I personally have rules for who I peer review for and– we’re doing this work for free, right? Technically, I know that the narrative in academia in particular is that this is part of your job. This is part of your paycheck. Well, these come in all the time, even when I’m not on contract, and this isn’t work that’s necessarily given to me in my daily job. So I challenge that assumption. I would say, if this is also my work then my job is not paying me enough. So I only peer review for open access journals. Because I know they’re not going to pay me. So my thought is: “Well, if you’re not going to pay me, then I want to know that I can access any of the things you publish for free.” So to me, that’s my payment.

Rene: That makes sense. And you know, Erin, I think prior to this. I didn’t have any hard and fast rules as it related to who I would engage with, and who I wouldn’t. A lot of it boils down to time. Did I have the time to commit to a proper review? Do I feel comfortable in the knowledge that I have to do right by this particular one? But post-pandemic…I really have begun to challenge some of these things. It’s a fact that for many academics it’s considered a requirement. It’s part of what we do, a part of our pay, and I don’t subscribe to that at all. And then just the sheer amount of money that these journals and publishing houses are making. So really, if the journal doesn’t fit some of those open access criteria here more recently, I refuse to engage in that process. The two pre-existing journals that I’ve agreed to serve on their board don’t subscribe to that. And I joined those prior to me reshaping my values and beliefs of the publishing industry. I think when those terms are done, I’ll be done.

Erin: One of the narratives that I’d like to challenge in this blog post, and honestly in academia as a whole is this idea that we don’t have this control. We definitely don’t have all the control, but we have control over where we submit and who we support. And I feel like we can make those rules for ourselves. I’m sure some people would say “I’m only gonna do the top tier journals.” As for me, some of my best experiences as a peer reviewer have been peer reviewing for mid-level, I guess you would say, journals. People at these other so-called levels– they’re publishing great stuff.

Rene: Yeah, I agree, and even looking at what all those metrics are, and how those are defined. I think I have really started to focus my time and energies and journals that better align with my values and beliefs and approaches towards work. 

Erin: Right. Another thing we were going to talk about was common problems in peer reviewing.

Rene: I think the system as a whole has some major issues. I get the lack of participation standpoint from individuals to engage as peer reviewers. Especially when I got an invitation that said, “I need this in 30 days.” And it was at the start of June when I was off my regular contract, but I was teaching 2 classes on a summer contract, and so it felt unfair and unrealistic. But I get what they’re doing, right? We want decisions rendered in a very timely fashion for the author. Particularly recognizing the implications from a promotion and continuing position standpoint. But, recognizing that publishing is a billion dollar industry, it’s difficult as faculty, emerging from the great resignation, to want to engage and say, “yes, I will do this in 30 days. I will read this 12,000 word manuscript, and read it in ways that most humans don’t engage with text” for nothing, right? To add to the knowledge community and this and that. But at the end of the day, when you look at the work faculty engage in and at how taxed we are as a group, it’s difficult to say yes to that timeline. I think we’re starting to find faculty who are pushing back more and saying. “Well, what’s my incentive?” I get that those researchers are getting whatever salary they’re getting. But it’s not commensurate with the amount of work that they’re doing in writing this publication. We know it’s not commensurate with the amount of time and effort and work that goes into evaluating this as a peer reviewer, and not just for one round of revisions, but maybe a second, or third! You’re working with a partner and an editor, and so you’ve married yourself to this document, for who knows how long and with what incentive? I think one of the most common issues is just simply the fact that this is coming off the backs of individuals who are engaging in a system that has one of the highest profit margins of any industry known to man. When you look at the amount of free labor throughout the entire system… I do think that one of the common problems is folks not wanting to engage in review and really needing to revisit incentives. Creating incentives beyond “this is what you should be doing,” or “this might be a good line item on your CV” if you want to move for promotion and tenure purposes. I think other common problems are just the timing of it all.  Finding realistic time to provide proper review in the midst of high teaching loads, possibly high advising loads, committee work. You know, this isn’t all we’ve committed ourselves to doing. And so I think that’s another kind of common problem with the industry and the process as a whole.

Erin: And librarians know what you’re talking about–the high cost, the high gains of the publishing industry. Librarians know it from a different perspective as well, which is that the cost of subscriptions for databases could have six figure price tags, for some of these expensive databases. Talking about turnaround times, we’re all doing too much, and librarians are also doing too much. Then maybe we need longer turnaround times for reviewing. But then, if you have an academic librarian or faculty member who is tenure track, what do we do with that person who really does need to get stuff out as fast as they possibly can? If they’re waiting 60 days and it’s a rejection, that’s 60 days it could have been somewhere else that might take it.

It even makes me think, do we need to be asking people to identify if they’re tenure track or not, if they’re on a timeline like that? Maybe we need to change tenure policies. If we alter how long people are gonna wait, we may have to alter what’s required of tenure. I believe we can set rules for ourselves. When I get a peer reviewer request, the first thing I consider is if I feel that I’m qualified to speak on the topic that the article is about, and I try to be as blunt as I can with myself about that so I’m doing justice to that article. Then the next question I have is: what am I doing in the next 2 weeks? If I don’t have time, I say: “no, I can’t do this.” Because I don’t want somebody waiting on me too long.

Rene: I think part of it is tackling it on two fronts and really looking at promotion and tenure as a whole, as it relates to scholarship. Also looking at the publishing industry and processes, I think both of those need to be revisited. Part of it is to be more strategic with who and how we engage in this work. I’ve seen some faculty post screenshots of messages that they sent back to editors when they’ve been invited to review, in which the potential peer reviewer says: “I’d love to engage in this work. What’s the incentive? How much will I be paid? What’s the honorarium?” Knowing that the answer is going to be nothing. And again, I don’t want to make it all about money, but I do think we need to do better, and we need to do right by peer reviewers and authors as well. I don’t know why we don’t question it a little bit more. 

A prime example: 3 years ago now, probably to the date, I get an email from an individual who doesn’t have access to an article that I wrote. In order to access it from the journal’s website, I think they would have had to pay $60. He said “I really don’t want to pay that amount to access your article. Would you be so kind to just share it with me?” And it blew my mind that he would have spent $60 on an article that I worked years on, literally, from rejection to rejection, to revisiting this, to the data collection. I mean this article, of all the things that I’ve worked on, is probably the one that means the most to me, and it pains me to hear him say that. And so I think one of the things that I wish I would have done back then is emailed the journal and said: “Hey! Where is my small percentage of this? Recognizing that you’re in this 1 billion dollar industry?” And so I do hope that peer reviewers begin to challenge the system in ways that they feel comfortable with so that we can continue to engage in doing top quality work, continue to have great work out in ways that are not constructed in manners that take advantage of those who feed and support the system.

Erin: Yeah. When you think about the cost of accessing these articles, that’s one of the reasons why I, a librarian, have a job right? Interlibrary loans are absolutely essential, otherwise you’re contacting authors all the time. That’s not feasible either. So library subscriptions and interlibrary loans are just a hugely important part of the process. And with academia and libraries in general, budgets are being cut everywhere. That has real implications for access. For me as a librarian and researcher and reviewer, the reason I focus on open access journals more than anything else is this thought that if I’m not leading this charge, how can I expect anybody else to do it? If I, as a librarian, don’t see the value in supporting open access over other kinds of access then how am I supposed to convince other faculty members at my university to support this, and to publish in these places? I feel like librarians have a lot of potential power in this arena that doesn’t always get used.

Rene: We’ve gathered data on this area. It’s interesting. The conversations we have with people who, at the end of whatever data we’ve gathered specific to publishing and the peer review system, say: “I never really thought of that. I didn’t feel like I was being exploited or taken advantage of.” And now that I’ve had this conversation. I very well might have, and didn’t really stop to think about what I was doing or how I was doing it. Or why I was doing it. 

Erin: When you have the best opportunity to think about how this is an exploitative system – I really think about tenure track librarians and tenure track faculty. When you have that kind of pressure on you where it’s that publish or perish mentality. So, publish or get out, find a new job. I always knew it was a problem, but I never knew how big of a problem it was until I was on this deadline. And then, all of a sudden, I was seeing things totally differently, the work I was being asked to do totally differently. So for me it was entering this process that really made me think like I need to contribute to this in a different way.

Rene: I couldn’t agree more. There are some who might take issue with creating a system that not necessarily incentivizes publishing, but better yet honors the work of peer reviewers and authors. I do believe we can live in an academic publishing world where some of the money that comes into the journal can go out in the form of stipends or honorariums for the work of peer reviewers and authors, you know. I think back to that article that the individual would have paid $60 for. Who knows who may have bought it? That a small percentage of that could have gone back, that we could create systems that are still structurally sound while also honoring the time and effort of the authors and the peer reviewers. I think the second money gets involved, individuals automatically think that the system is somewhat rigged, and I genuinely believe that we can create environments and systems and processes where we can pay peer reviewers for the time and effort and the commitment that they make in reading these documents in ways that most people don’t read documents and connect themselves to it, whether it’s one revision or 5 revisions. 

Erin: I would just love it if these subscription journals said, “Hey. We’re gonna call you for–I don’t know. I’m just throwing this number out there– We’re gonna call you this year for 4 different articles for review. And if you can commit to 4 articles for review this year, we’re going to give you a subscription to the journal for the year.” I would even do that, because that’s great. That’s a journal I probably also love reading. I would do it even for something like that.

Rene: I think there are other things that journals can do to incentivize or encourage participation in reviewer activities. I was asked to peer review 2 book chapters a couple of years back, and in doing it they were going to give me a copy of the book, and they were going to provide me a credit of $200 to the publishing house to get other books from within them. And probably it didn’t cost them a whole lot, but meant a great deal to me. And so I joyfully said yes. I think there are smart strategic things that folks would love and would make them feel valued. And I think right now there’s this post pandemic feeling in the Academy where folks just aren’t feeling that their value and their worth is being met. And I think these are things that the publishing industry should really consider. And these are things that I think scholars who engage in this work should demand. And I don’t think there’s anything wrong with that.

Erin: Yeah, I don’t understand why it’s not something more presses do. The more people have access to stuff, tends to equal more citations which raises the esteem and prestige of the press, and the authors as well. I’m sitting here thinking what a missed opportunity on the part of these presses, because how many academic books come out every year that get almost no readers? If you got that book and it’s good, you’re probably going to cite that book at some point in the future in your research, if it’s at all relevant. I certainly do! And if these publishers give you credit, you’re gonna purchase books with that credit that are relevant to your research. Which means you’re going to be citing those books which is going to raise the citation metric of all those books. And that press is going to look better and better in so many ways. 

There were a few things we did not consider during the conversation snippet you see here, that we talked about later. One of those is to consider whether or not peer reviewers get any training on the role at the journal, or what other kinds of support exist for peer reviewers. Rigorous training on how to be a peer-reviewer for a specific journal seems to be very rare, which is disappointing. However, most journals at least offer peer reviewers rubrics or lists of questions to answer to help guide their reviews. Though this is a great help as it makes sure reviewers are considering some of what the editors want, the complexity is sometimes up to the reviewer themselves, and rubrics can be vague where we might wish for more guidance. All of this (training, expectations, opportunities for support, etc.) is, however, something that could be tackled as peer reviewers make the shift into editor positions. 

Another thing we talked about is how we can support writers (one of Erin’s favorite things) as peer reviewers. It’s not uncommon that some of the feedback from peer reviewers is confusing. One way to create a supportive environment could be to make, through some anonymous (or otherwise) process, the reviewer available for clarification purposes on their specific feedback. There are some journals where being a peer reviewer is more of a mentor position for writers than a strictly peer reviewing gig. If supporting writers is one of your goals, one of these journals could be a good fit for your peer reviewing skills (additionally, authors should take note of journals that offer these sorts of opportunities if they’re new to the process or maybe have a difficult article they’re trying to write). Several times Erin has come across an article that had a lot of problems, possibly too many to publish without significant revisions, and she reached out to the editors of the journal offering to work with the authors more closely as they revise. These were articles she felt very strongly had potential to make an important mark in our field, or that filled an obvious gap in research. For Erin, it was worth the extra time to support these writers and their research. 

We invite Librarian Parlor readers to respond in the comments section with names of journals they’ve had positive experiences with as far as supporting writers and peer-reviewers is concerned (Erin, for example, has had great experiences on both sides with the teams at the Journal of New Librarianship and the International Journal of Librarianship). Remember, when considering how to make that transition from writer to peer-reviewer, we really can make our own rules. And there is no one right answer, just the right answer for you. 

---

_{Photo by Icons8 Team on Unsplash}

---

_{This work is licensed under a Creative Commons Attribution 4.0 International License.}

---

_{The expressions of the writers do not reflect anyone’s views but their own.}