Academic journals are increasingly asking authors to use transparent reporting practices to “trust, but verify” that outcomes are not being reported in a biased way and to enable other researchers to reproduce the results. To implement these reporting practices, most journals rely on the process of peer review — in which other scholars review research findings before publication — but relatively few journals measure the quality and effectiveness of the process.
In a commentary published July 20 in the journal Science, lead author Carole Lee and co-author David Moher identify incentives that could encourage journals to “open the black box of peer review” for the sake of improving transparency, reproducibility, and trust in published research. Lee is an associate professor of philosophy at the University of Washington; Moher is a senior scientist at The Ottawa Hospital and associate professor of epidemiology at the University of Ottawa.
Lee and Moher see this as a collective action problem requiring leadership and investment by publishers.
“Science would be better off if journals allowed for and participated in the empirical study and quality assurance of their peer review processes,” they write. “However, doing so is resource-intensive and comes at considerable risk for individual journals in the form of unfavorable evidence and bad press.”
To help journals manage the reputational risk associated with auditing their own peer review processes, Lee and Moher suggest revising the Transparency and Openness (TOP) Guidelines, a set of voluntary reporting standards to which 2,900 journals and organizations are now signatories. These guidelines were published in Science in 2015 by a committee of researchers and representatives from nonprofit scientific organizations, grant agencies, philanthropic organizations and elite journals.
Lee and Moher suggest adding a new category to the TOP guidelines “indicating a journal’s willingness to facilitate meta-research on the effectiveness of its own peer review practices.” With these, journals can choose which tier or level they take on. Higher levels of transparency would involve higher risk.
- For the lowest tier, journals would publicly disclose whether they are conducting internal evaluations of peer review, in which they are able to retain the study results for internal use.
- At the middle tier, journals would disclose the results of their internal evaluations of peer review, but could maintain flexibility in how they report their results for external use. For example, results could be aggregated across several journals to reduce risk to any single journal.
- At the upper tier, journals could agree to relinquish data and analyses to researchers outside their institution for third-party verification. This is an option, Lee and Moher write, “that might appeal especially to publishers with fewer resources, as it places the financial burden on those conducting the meta-research.” Journals conducting their own analyses could preregister their study designs then deposit their data publicly online.
By agreeing to these more stringent guidelines, the authors write, publishers and journals would have the chance to legitimize and advertise the relative quality of their peer review process in an age when predatory journals, which falsely claim to use peer review, continue to proliferate.
“Illegitimate journals are becoming a big problem for science,” said Moher. “True scientific journals can distinguish themselves with transparence about their peer review processes.”
Investing in research on journal peer review will be costly, they agree. Lee and Moher suggest that large experimental studies are needed to judge the effectiveness of different web-based peer review templates to enforce reporting standards, and of ways one might train authors, reviewers and editors to use such tools and evaluate research.
Also needed, they say, are ways to detect shortcomings in statistical and methodological reporting on a research paper, and to understand how the number and relative expertise of peer reviewers can improve assessment.
The largest publishers, whose profit margins compete with those of pharmaceutical and tech giants, can afford to invest in the requisite technology and resources needed to carry out these audits, the researchers say.
“Publishers should invest in their own brands and reputations by investing in the quality of their peer review processes,” said Lee. “Ultimately, this would improve the quality of the published scientific literature.”
The Latest on: Journal peer review
- AMCP's Journal of Managed Care & Specialty Pharmacy (JMCP) Adds Two Section Editors as Publication Marks 25th Anniversaryon January 23, 2020 at 8:17 am
Their addition also ensures JMCP will remain the go-to, peer-reviewed journal for researchers seeking a fast and fair author experience." The average time it takes JMCP to make a first decision on ...
- Stuart J. Heckman To Helm The Journal Of Financial Planningon January 22, 2020 at 10:52 am
Heckman, who will serve a two-year term, will review and evaluate research manuscripts to determine if they proceed to the Journal's double-blind peer review process. He will also provide guidance to ...
- Viticulture Data Journal: Non-conventional papers foster open science & sustainabilityon January 21, 2020 at 8:35 am
editors and other invited contributors enjoy the convenience of working within the same consolidated online file all the way from the authoring and peer review stages to copy editing and publication. ...
- Increasing physical activity by four legs rather than two: systematic review of dog-facilitated physical activity interventionson January 20, 2020 at 2:22 pm
Design Systematic review. Data sources We carried out literature searches to August 2019 using six common databases. Eligibility criteria Studies included published papers in peer-reviewed journals ...
- WiFi Cameras Market Latest Review: Know More About Industry Gainerson January 20, 2020 at 10:36 am
WiFi Cameras Comprehensive Study by Type (Security & Surveillance Camera, Digital Camera with Wi-Fi, Car Camera, Sports Camera), Application (Home Security, Consumer Electronics, Sports Enthusiasts, ...
- GRUR Int. – English Relaunch of a Well-Established Law Journalon January 20, 2020 at 2:31 am
Articles will be selected based on an independent peer-review process. A pioneer among international journals in the field of IP and competition law, the "Gewerblicher Rechtsschutz und ...
- Stuart Heckman Named Academic Editor for Journal of Financial Planningon January 15, 2020 at 5:17 pm
... research manuscripts for the Journal’s double-blind peer review process, and providing leadership to staff. Heckman will continue his work at Kansas State University as an associate professor and ...
- Cybersecurity Skills Journal (CSJ): Practice and Research Call for Abstractson January 13, 2020 at 11:00 pm
The Cybersecurity Skills Journal (CSJ): Practice and Research is calling for every and anyone to submit abstracts, meaning just ideas, thoughts or experiences, related to workforce capabilities ...
- Predatory-journal papers have little scientific impacton January 13, 2020 at 5:58 am
Predatory journals are those that charge authors high article-processing fees but don’t provide expected publishing services, such as peer review or other quality checks. Researchers and publishers ...
- Opinion: Exorcising Ghostwriting from Peer Reviewon January 12, 2020 at 9:04 pm
As defined in that paper, peer review ghostwriting occurs when scientists hand over manuscripts that they have agreed to review for journal editors to graduate students or postdocs in their research ...
via Google News and Bing News