Can AI fix the peer review bottleneck? Top journals weigh in

As AI reshapes the peer review process, top medical journals wrestle with balancing innovation and ethical integrity while redefining standards for the future of scientific publishing.

Study: Use of Artificial Intelligence in Peer Review Among Top 100 Medical Journals. Image Credit: Anastasia Markeeva / Shutterstock.com

A recent study published in JAMA Network Open explores attitudes towards the use of artificial intelligence (AI) in medical journals.

A sudden surge in medical research publishing

A significant increase in medical research publications has put increasing pressure on the peer-reviewing process, which is slowing due to the lack of enough experts to review articles. Repeated reviews of rejected manuscripts have questioned the fairness and efficiency of the existing peer-reviewing system.

Generative AI (GenAI) refers to deep-learning models that are capable of reviewing raw data as their training set to generate new work that is similar to the original data. The application of AI, particularly GenAI, has the potential to accelerate the peer review process; however, the use of AI in publications varies across journals, depending on their existing policies.

As AI continues to advance, it is crucial to perform a comprehensive survey of medical journals’ guidance to understand how AI-assisted peer review is perceived and the reasons behind this conviction.

About the study

The current study obtained relevant data from Scimago.org. It selected the top 100 medical journals to evaluate their guidance on using AI in peer-review processes.

Between June 30, 2024, and August 10, 2024, the websites of the selected journals were searched for AI-related policies. If the journal did not offer detailed AI guidance but provided a link to the publisher’s guidelines, the latter was used as a substitute.

Study findings

Approximately 78% of medical journals provided comprehensive guidance on AI use in their peer review processes. Interestingly, 59% of these journals have strictly prohibited AI use, whereas the remaining 41% of journals allowed its use only if confidentiality was maintained and authorship rights were respected.

Journals with editorial offices in the United States or Europe were more likely to prohibit AI use than internationally based medical journals. Mixed publishers were also associated with a greater tendency to prohibit AI use.

Approximately 91% of the journals that provided guidance on AI use forbade uploading manuscript-related content to AI. However, 32% of these journals allowed restricted AI use with disclosure in review reports.

About 27% and 47% of journals referred to AI tools like large language models and chatbots, respectively. Certain medical journals also provided preferences in AI use; for example, Springer Nature favor limited AI use, whereas Cell Press and Elsevier altogether banned AI use. Many journals were reluctant to use GenAI due to its inherent problems that could lead to biases and confidentiality breaches.

Confidentiality concerns have been attributed as the main reason for not permitting or limiting AI use in medical journals. In the future, a similar study on guidelines provided by low-ranking journals should be performed to assess and compare their AI policies to those described in the current study.

Study interpretation

The current study highlights the critical attitudes of top medical journals towards AI-assisted peer review. For example, journals that permit limited AI use had variable disclosure standards in crucial areas of innovation and reproducibility. However, these journals also fail to provide comprehensive guidance on reference management.

The lack of well-organized and often scattered AI-related guidance makes it difficult for reviewers to access and comply with these standards and contributes to existing confidentiality issues. Therefore, it is important that editors clarify their instructions for future article submissions.

Benefits of AI use in peer review

The survey performed in the current study revealed that many researchers using AI in their work support editorial and review processes that also utilize AI. A higher familiarity with AI and its technical capabilities could improve its use in peer review processes in the future.

The safe and ethical use of AI has the potential to significantly improve innovation, productivity, and high-quality peer reviews. However, the authors warn that AI should never be used to replace human peer review.

Continuous monitoring and regular assessment of AI’s impact are essential for updating guidance, thereby maintaining high-quality peer review.”