How to Review a Journal Article

The Review Process

You've been selected as an ACTFL article reviewer, congratulations! In order to help you with this important task, we've collected nine things you should consider as you examine the manuscript and write your review:

Look for the “intellectual plot-line” of the article. You can do this from first skimming through the manuscript and then giving it a once-over read. As you do this, ask the five major questions that are central to the research review process:

1. What do the researchers want to find out?
2. Why is that important to investigate or understand?
3. How are the researchers investigating this? Are their research methods appropriate and adequate to the task?
4. What do they claim to have found out? Are the findings clearly stated?
5. How does this advance knowledge in the field?
6. How well do the researchers place their findings within the context of ongoing scholarly inquiry about this topic? Look at the organization of the article. Can you find answers to the above questions quickly and easily? Can you trace the logic of investigation consistently from the opening paragraphs to the conclusion?
7. Then go back to the opening paragraphs of the article. Are the research questions specifically stated? Is it clear what the authors want to find out? Do they make the case that this is an important area for research inquiry?
8. The next section is usually a review of the existing research literature on this topic. Do the authors present a convincing line of argument here—or does it appear that they are just name-dropping (citing sources that may be important, without a clear underlying logic for how they may be important)? Do the authors focus on ideas, or merely on discrete facts or findings? Have they given sufficient attention to theory—the cumulative attempts at prior explanations for the questions they are investigating? Are the research questions or hypotheses clearly derivative of the theory and the literature review? In short: How well do the authors set the stage for the research problem they are reporting?
9. The methods and procedures section is usually next; and this is where neophyte reviewers often start (unwisely) to sharpen their knives. The selection of methods by which the researchers collect data always involve compromises, and there are few studies that cannot be criticized for errors of commission or omission in terms of textbook criteria for research design and data collection procedures. You could focus on three questions here:
   1. Do the authors clearly describe their research strategies? Do they present sufficient detail about the sample from which they have collected data; the operationalization of measures they have attempted to employ; and the adequacy of these measures in terms of external and internal validity? In addition, there should be no surprises here: The measures should be clearly matched to the research questions or the hypotheses.
   2. Are their choices of methods adequate to find out what they want to find out in this study? Would other methods provide a substantial improvement; if so, would employing these methods be feasible or practical?
   3. Do they provide some justification for the methods they have chosen? Does this appear to be adequate? The section presenting research results is surely the heart of the article— though not its soul (which the reader should find in the opening paragraphs and in the discussion section).
      1. Does the results section tell a story—taking the reader from the research questions posed earlier to their answers in the data? Is the logic clear?
      2. Are the tables and figures clear and succinct? Can they be “read” easily for major findings by themselves, or should there be additional information provided? Are the authors’ tables consistent with the format of currently accepted norms regarding data presentation?
      3. Do the authors present too many tables or figures in the form of undigested findings? Are all of them necessary in order to tell the story of this research inquiry; or can some be combined? Remember that tables and figures are very expensive (from the standpoint of the journal) and that undigested data obscure rather than advance the cumulative development of knowledge in a field.
      4. Are the results presented both statistically and substantively meaningful? Have the authors stayed within the bounds of the results their data will support?

Reviewers might consider four questions here:

The discussion section is where the authors can give flight to their findings, so that they soar into the heights of cumulative knowledge development about this topic—or crash into the depths of their CV’s, with few other scholars ever citing their findings. Of course few research reports will ever be cited as cornerstones to the development of knowledge about any topic; but your review should encourage authors to aspire to these heights. Consider the following as you evaluate their discussion section:

1. Do the authors present here a concise and accurate summary of their major findings? Does their interpretation fairly represent the data as presented earlier in the article?
2. Do they attempt to integrate these findings in the context of a broader scholarly debate about these issues? Specifically: Do they integrate their findings with the research literature they presented earlier in their article—do they bring the findings back to the previous literature reviewed?
3. Have they gone beyond presenting facts—data—and made an effort to present explanations—understanding? Have they responded to the conceptual or theoretical problems that were raised in the introduction? This is how theory is developed.
4. Do the authors thoughtfully address the limitations of their study?

The writing style is important. Consider the three guidelines for successful communication—to be clear, concise, and correct---and whether the authors have achieved it:

1. Is the writing clear? Do the authors communicate their ideas using direct, straightforward, and unambiguous words and phrases? Have they avoided jargon (statistical or conceptual) that would interfere with the communication of their procedures or ideas?

2. Is the writing concise? Are too many words or paragraphs or sections used to present what could be communicated more simply?

3. Is the writing correct? Too may promising scientists have only a rudimentary grasp of grammar and punctuation that result in meandering commas, clauses in complex sentences that are struggling to find their verbs, and adjectives or even nouns that remain quite ambiguous about their antecedents in the sentence. These are not merely technical issues of grammar to be somehow dealt with by a copy-editor down the line. Rather they involve the successful communication of a set of ideas to an audience; and this is the basis of scholarship today.

Your evaluation to the editor: Should this paper be (a) rejected for this journal? (b) or does it show sufficient promise for revision, in ways that you have clearly demonstrated in your review, to encourage the authors to invest weeks and months in revision for this journal? Your bottom-line advice to the editor is crucial. Make a decision; state it clearly (in your confidential remarks to the editor on the page provided). Remember that only a few of the articles submitted to a journal will result in publication. Rates vary from 5% to 25% of initial submissions.

Some reasons to reject a manuscript: (a) The research questions have already been addressed in prior studies; (b) the data have been collected in such a way as to preclude useful investigation; (c) the manuscript is not ready for publication—incomplete, improper format, or error-ridden. Most rejected articles do find a home in other journals. Don’t tease authors with hopes for publication in this FLA if you feel it is not likely.

Good Reviews and Bad Reviews

A good review is supportive, constructive, thoughtful, and fair. It identifies both strengths and weaknesses, and offers concrete suggestions for improvements. It acknowledges the reviewer’s biases where appropriate, and justifies the reviewer’s conclusions. A bad review is superficial, nasty, petty, self-serving, or arrogant. It indulges the reviewer’s biases with no justification. It focuses exclusively on weaknesses and offers no specific suggestions for improvement.