Can LLMs Review Scientific Papers? A Case of Hallucination and Grounding Failure in Journal Peer Review

Keywords: large language models, peer review, hallucination, grounding failure, research integrity, retrieval-augmented generation, AI-assisted review

1. Introduction

Artificial intelligence systems have shown increasing ability to perform specialized tasks across domains, including domain-specific prediction, classification, and decision-support tasks [9-13]. However, successful automation in constrained tasks does not imply that AI systems can safely perform responsibility-bearing scholarly judgments. Peer review is not merely a text-generation task. It is an act of scholarly judgment. A reviewer is expected to read a manuscript, understand its claims, evaluate its evidence, identify limitations, and provide justified criticism under their own responsibility. Large language models (LLMs), however, make it increasingly easy to generate fluent, formal, and plausible review reports with minimal engagement with the manuscript.

This creates a serious problem for scholarly publishing. A review may sound professional while failing to remain grounded in the paper it evaluates. In such cases, the issue is not simply poor writing or harsh criticism. The issue is that an official editorial evaluation may contain claims that are unsupported by, or directly contradicted by, the manuscript.

The problem is closely related to the broader issue of hallucination in LLMs. Prior work has shown that retrieval or grounding can improve factuality in knowledge-intensive generation. Retrieval-Augmented Generation (RAG) combines parametric model knowledge with retrieved non-parametric evidence and was shown to generate more specific and factual language than a parametric-only baseline on knowledge-intensive tasks [1]. Retrieval-enhanced language models such as RETRO similarly condition generation on retrieved text chunks from a large corpus, showing that external retrieval can improve language modeling performance and factual grounding while reducing reliance on model parameters alone [2]. Self-RAG further proposed adaptive retrieval and self-reflection to improve factuality and citation accuracy in long-form generation [3]. Browser-assisted systems such as WebGPT also emphasize that collecting references during generation can make factual accuracy easier to evaluate [4].

This literature suggests a plausible hypothesis for peer review: if LLMs are used to assist review writing, grounding review claims in the source manuscript may reduce hallucination-like errors. A useful review should not merely say that “the experimental design is insufficient” or “the dataset is unclear.” It should identify what is missing, where the issue occurs, why it matters, and how the manuscript can be improved. In other words, peer-review criticism should be attributable to the manuscript.

However, grounding is not a complete solution. Hallucinated citations and fabricated references have been empirically documented in academic contexts, and such outputs can appear legitimate at first glance [5]. This is particularly concerning in peer review, because review reports can influence editorial decisions, publication timelines, and the scientific record.

Publishers have also begun to treat AI use in peer review as an issue of confidentiality, ethics, and accountability. Nature Portfolio states that peer reviewers are accountable for the accuracy and views expressed in their reports and warns that generative AI tools can produce false information [6]. SAGE states that reviewers who inappropriately use generative AI tools to generate review reports will not be invited to review and that such reports will not be included in final decisions [7]. Recent case-based work has also begun to document first-person experiences with suspected AI-generated peer reviews [8].

This paper presents a case study of hallucination-like and ungrounded evaluation in formal journal peer review. The case concerns three peer-review reports received during an actual submission to a Korea Citation Index (KCI) candidate journal. One review made multiple claims that important manuscript elements were missing, although the manuscript explicitly contained those elements. The author reported these concerns to the editor; after editorial consideration, the problematic review was not relied upon in the final decision, and the manuscript was subsequently accepted and published.

The goal of this paper is not to prove that the problematic review was generated by an LLM. Rather, the goal is to analyze a failure mode that is highly relevant to LLM-assisted reviewing: fluent but ungrounded evaluation. This paper asks three questions:

1. What kinds of grounding failures appeared in the problematic review?

2. How did the problematic review differ from two more grounded reviews of the same manuscript?

3. Does source-grounded LLM review reduce the same type of false missing-content claims?

The central argument is that the issue is not merely whether AI was used. The issue is whether the review was grounded, verified, disclosed where required, and responsibly owned by the human reviewer.

2. Case Description

2.1 Review context

This paper analyzes three formal peer-review reports received during an actual submission to a Korea Citation Index (KCI) candidate journal. The manuscript was a technical paper in applied artificial intelligence. To reduce identifiability, the journal name, manuscript title, system name, detailed application domain, author identity in the original submission, reviewer identities, editorial metadata, submission dates, publication date, volume, issue, page information, and bibliographic links are omitted.

Not all technical information is omitted. In a technical case study, complete removal of experimental details would make the grounding analysis impossible to evaluate. Therefore, non-identifying methodological metadata are retained where necessary, including hardware, software frameworks, model configuration, parameter count, and evaluation design. The case is described only at the level necessary to analyze whether review comments were grounded in the manuscript’s own technical content.

2.2 The three reviews

The manuscript received three independent reviews. They are referred to as Review A, Review B, and Review C.

Review A recommended publication after revision. It recognized the manuscript’s main contribution and requested additional comparative evaluation, broader test conditions, clearer visual explanation of results, and expanded discussion of limitations and applications. The comments were concise but directly connected to the manuscript.

Review B recommended revision and re-review. It summarized the manuscript in broad terms and requested revisions related to redundancy, experimental detail, data description, model parameters, computational setup, result analysis, comparison with previous methods, terminology, figures and tables, limitations, conclusion, references, and proofreading. Some of these requests were reasonable. However, several claims about missing methodological and experimental details were contradicted by the manuscript.

Review C recommended acceptance. It accurately summarized the manuscript’s motivation, contribution, and practical relevance. It also identified a concrete limitation that was actually discussed in the manuscript: the method can produce residual artifacts under certain conditions. Review C suggested that the manuscript would be strengthened by discussing this limitation and possible remedies more fully.

2.3 Editorial follow-up

After receiving the reviews, the author contacted the editor to report concerns that Review B appeared insufficiently grounded in the manuscript and potentially AI-generated or AI-assisted. The concern was not based solely on writing style. It was based on multiple review claims that contradicted explicit content in the manuscript, including claims about missing validation data description, computational environment, model configuration, and methodological details.

Following editorial consideration, Review B was not relied upon in the final assessment. The manuscript was subsequently accepted and published. This outcome does not prove that Review B was generated by an LLM. However, it shows that the review was sufficiently unreliable to require editorial intervention. The case therefore illustrates not only a textual failure mode, but also the importance of editorial mechanisms for identifying and correcting ungrounded review reports.

3. Analytical Framework

This paper defines grounding as the degree to which a review claim is supported by specific evidence in the manuscript. A grounded review does not merely state that “the experimental details are insufficient.” It identifies what detail is missing, where the issue occurs, why the omission matters, and how the manuscript can be improved.

The term hallucination is used in a functional sense. In this paper, a hallucinated review claim is a claim that is fluent and plausible but contradicted by the source manuscript. The term is not used as proof that the review was generated by an LLM.

3.1 False missing-content claim

A false missing-content claim occurs when a review states that some information is absent even though the manuscript explicitly contains it.

Example form: “The computational environment is not described.”

This is a grounding failure if the manuscript explicitly states the hardware and software environment.

3.2 Methodological hallucination

A methodological hallucination occurs when a review misrepresents the method, claims that methodological components are absent when they are present, or criticizes the paper on the basis of an incorrect reconstruction of the proposed approach.

3.3 Evaluative hallucination

An evaluative hallucination occurs when a review makes broad judgments such as “insufficient,” “unclear,” or “unsupported” without connecting the criticism to manuscript evidence. The claim may be plausible in generic terms but remains weak as peer review because it is not source-grounded.

3.4 Template-like review generation

Template-like review generation occurs when a review lists generic revision requests that could apply to many manuscripts: improve the abstract, define terminology, expand references, clarify figures, discuss limitations, and proofread. These comments may be reasonable individually, but if they are not tied to manuscript-specific evidence, they provide limited editorial value.

3.5 Accountability failure

An accountability failure occurs when a reviewer submits unverified generated or semi-generated evaluative text as an official peer-review report. This is not only a linguistic problem. It is a responsibility problem. A review submitted under a reviewer’s name is an official scholarly judgment, regardless of whether AI tools were used during drafting.

4. Methods

4.1 Manual claim-level audit

The author manually compared the substantive claims in Review B against the manuscript. Each claim was classified as one of the following:

1. Supported: clearly supported by the manuscript.

2. Partially supported: has some basis, but is vague, overstated, or incomplete.

3. Contradicted: claims something is missing or incorrect, although the manuscript explicitly contains the relevant information.

4. Unsupported/generic: not clearly tied to the manuscript.

5. Not assessable: cannot be evaluated from the manuscript alone.

The manual audit was treated as the primary evidence in this paper.

4.2 Review comparison

Reviews A, B, and C were compared on the following dimensions:

• manuscript grounding

• specificity

• factual accuracy

• actionability

• genericity

• false missing-content claims

• suitability as a basis for editorial decision-making

4.3 Exploratory source-grounded LLM review

The author conducted an exploratory source-grounded LLM review using a manuscript-only prompt in a notebook-style source-grounded LLM system. The uploaded source consisted of the anonymized manuscript PDF only. Prior reviewer reports were not included in the notebook to avoid contamination.

The prompt asked the system to act as a peer reviewer and evaluate the manuscript in terms of topic suitability, abstract clarity, literature review, methodology, experimental design and results, references, originality, and academic contribution. The output was then compared with Review B.

This experiment was not designed to prove that a source-grounded LLM can replace a human reviewer. Rather, it was used to examine whether a source-grounded LLM output under a simple review prompt would reproduce the same type of false missing-content claims observed in Review B.

5. Results

5.1 Comparison of the three formal reviews

Review A was concise but grounded. It identified areas where the manuscript could be strengthened, including comparative evaluation, broader testing, clearer presentation of visual results, and expanded limitations.

Review C was also grounded. It accurately captured the manuscript’s central contribution and identified a limitation that was actually present in the manuscript. Its recommendation was positive, but it still provided a concrete improvement request.

Review B differed from the other two reviews. Although it was fluent and professionally formatted, several of its claims were contradicted by the manuscript.

5.2 False missing-content claims in Review B

Table 2 summarizes the claim-level audit of Review B. The most serious grounding failures were false missing-content claims. Review B stated or implied that the manuscript lacked sufficient information about validation data, hardware, software frameworks, model parameters, and methodological details. However, as shown in Table 2 and Appendix A, the manuscript explicitly contained these elements.

The most serious issue was not that Review B gave negative feedback. Negative feedback is essential to peer review. The issue was that some negative feedback was based on claims of absence that were factually inconsistent with the manuscript. This is a high-risk review failure because the report appears reasonable while its factual premise is wrong.

5.3 Source-grounded LLM review result

The source-grounded LLM review was substantially more aligned with the manuscript than Review B. Unlike Review B, it recognized that the manuscript specified the validation data source, hardware environment, software frameworks, model configuration, algorithmic pipeline, and comparative model-size information. It did not claim that the validation data, computational environment, or model parameters were absent.

Its major criticisms were mostly grounded in real weaknesses of the manuscript.

However, the source-grounded LLM output was not error-free. It misreported the number of references and overstated one technical limitation. These errors were smaller than the false missing-content claims in Review B, but they show that grounding does not eliminate hallucination, counting errors, or interpretive overreach.

5.4 Review B versus source-grounded LLM output

Grounding changed the failure mode. Review B produced several false missing-content claims. The source-grounded LLM review did not reproduce those major errors, but still produced local inaccuracies and overstatements. This distinction is central to the paper’s argument.

6. Discussion

6.1 Grounding improves review validity

The comparison suggests that grounding substantially improves the validity of LLM-assisted review. The source-grounded LLM review was more closely aligned with the manuscript than Review B. It recognized the validation data, hardware/software environment, model configuration, algorithmic procedure, and comparison information. Its major criticisms focused on genuine weaknesses such as missing runtime benchmarks, limited quantitative evaluation, and insufficient discussion of mitigation strategies.

This finding is consistent with prior work on retrieval-augmented and source-grounded language generation. RAG was introduced to combine parametric knowledge with retrieved external evidence and improve factuality in knowledge-intensive generation [1]. Retrieval-enhanced models have similarly shown that retrieved text can improve language generation by providing external grounding beyond parametric memory [2]. Self-RAG further showed that adaptive retrieval and self-reflection can improve factuality and citation accuracy [3]. In peer review, the equivalent principle is that substantive review claims should be auditable against the manuscript.

6.2 Grounding does not eliminate hallucination

The same source-grounded output still contained factual errors and overstatements. The incorrect reference count is a small but clear hallucination. The overstated limitation also shows that a grounded system can still misinterpret or exaggerate source material.

This is important because grounding can create a false sense of safety. A citation, source link, or manuscript reference does not automatically make a claim correct. A review can point to a relevant section while still misreading it. Therefore, grounding should be understood as a necessary but insufficient safeguard. This conclusion is consistent with prior evidence that LLM outputs can contain fabricated or erroneous academic citations even when the surrounding text appears plausible [5].

6.3 The central problem is unverified judgment

This case does not support a simple anti-AI conclusion. Rather, it shows that ungrounded or unverified review text is dangerous, whether written by a human, an LLM, or a human using an LLM. The central issue is not merely whether AI was used. The central issue is whether the review was grounded, verified, and responsibly owned.

Review B was problematic because it inserted false claims into the editorial process. If such claims had been accepted without checking, the decision could have been influenced by information that was inconsistent with the manuscript. The author’s intervention and the editor’s decision to discount the problematic report prevented this failure from determining the final outcome.

6.4 AI assistance versus AI delegation

A distinction should be made between AI assistance and AI delegation.

AI assistance may include language polishing, organizing reviewer notes, checking grammar, or helping formulate already-verified comments. AI delegation occurs when the reviewer transfers substantive scholarly judgment—novelty assessment, methodological criticism, rejection rationale, or revision requirements—to an AI system without verification.

The latter is ethically problematic. Peer review is submitted under a reviewer’s authority. If a reviewer uses AI to help draft a report, the reviewer remains responsible for every claim. A model cannot answer author rebuttals, defend methodological criticisms, or accept accountability for false claims.

6.5 Implications for editorial policy

This case suggests several practical safeguards.

First, major criticisms should be required to cite specific manuscript sections, figures, tables, or passages. A review that claims “the computational environment is unclear” should identify where the relevant information should have been described and confirm that it is not described elsewhere.

Second, editors should treat generic checklist-style reports as low-information unless they are supported by manuscript-specific evidence.

Third, reviewers who use AI tools should disclose such use according to journal policy and should not upload confidential manuscripts into external tools unless permitted by the journal and compatible with confidentiality obligations. This aligns with publisher policies emphasizing reviewer accountability and confidentiality in relation to AI tools [6,7].

Fourth, authors should be allowed to contest reviews at the claim level. The appropriate response to an ungrounded review is not merely “the review is unfair,” but “this review claim is contradicted by the following manuscript passage.”

7. Recommendations

The findings of this case study suggest that responsible AI-assisted reviewing requires safeguards at multiple levels. The central issue is not only whether AI tools are used, but whether the resulting review claims are verified, attributable to the manuscript, and owned by the reviewer. Because peer review can influence editorial decisions and the scientific record, unsupported or generic criticism should not be treated as equivalent to manuscript-grounded scholarly evaluation.

For reviewers, the most important requirement is verification. Reviewers should not submit AI-generated or AI-assisted review text unless every substantive claim has been checked against the manuscript. In particular, reviewers should avoid stating that information is missing unless they have examined the full manuscript and confirmed that the information is not present. A distinction should also be maintained between information that is genuinely absent and information that is present but insufficiently emphasized, poorly organized, or inadequately discussed. These categories require different forms of criticism and different revision requests.

If reviewers use AI tools during the review process, such use should be disclosed according to the journal’s policy. AI tools may assist with language polishing, organization, or summarization of already verified notes, but they should not be used to delegate substantive scholarly judgment. The reviewer remains responsible for the accuracy, fairness, and implications of the submitted report.

Editors also have an important gatekeeping role. Major methodological criticisms should be expected to refer to specific manuscript evidence, such as sections, figures, tables, experimental descriptions, or passages. Generic checklist-style reports should be treated as low-information unless they are supported by manuscript-specific observations. Editors should also provide a mechanism through which authors can report review claims that are contradicted by the manuscript. Such a process would help distinguish ordinary disagreement from factual review errors.

Journals should establish explicit policies for AI use in peer review. These policies should address confidentiality, disclosure, reviewer accountability, and the permissibility of uploading manuscripts to external AI systems. Clear policies are necessary because the ethical risk lies not only in text generation itself, but also in the possibility that confidential manuscripts may be processed by tools outside the journal’s control.

Authors, meanwhile, should respond to ungrounded reviews at the claim level. Rather than simply arguing that a review is unfair, authors should map disputed review claims to specific manuscript evidence. This makes the rebuttal more verifiable and allows editors to evaluate whether the review criticism is grounded or contradicted.

Finally, AI-tool developers should design review-assistance systems that encourage claim-level attribution and uncertainty reporting. A responsible review-support system should not merely generate fluent criticism. It should make it easier to trace each criticism back to manuscript evidence, distinguish absence from insufficiency, and flag claims that require human verification.

8. Limitations

This study has several limitations.

First, it is based on a single manuscript and three reviews. It cannot estimate the prevalence of AI-generated or ungrounded peer review.

Second, the study does not prove that Review B used an LLM. The analysis is based on manuscript-review mismatch and review quality, not forensic AI detection.

Third, the reviewed manuscript itself had real weaknesses. For example, typographical errors were present, quantitative runtime benchmarks were absent, and limitations could have been expanded. Therefore, the purpose of this paper is not to argue that all criticism was invalid.

Fourth, the source-grounded LLM experiment was exploratory. It used one manuscript, one tool, and one prompt condition. The result should be treated as an illustrative comparison, not as a general benchmark of LLM review quality.

Fifth, anonymization limits external verifiability. To reduce identifiability, the journal name, manuscript title, detailed application domain, system name, bibliographic metadata, and editorial metadata are omitted. However, hardware, software framework, model-size, and parameter-count information are retained where necessary, because removing all technical details would make the grounding analysis unverifiable.

9. Conclusion

Can LLMs review scientific papers? Maybe not—at least not without grounding, verification, and human accountability.

This case study shows that a peer review can appear polished and professional while making claims that are poorly grounded in the manuscript. The problematic review claimed that validation data, hardware/software environment, model configuration, and methodological details were missing or insufficient, although the manuscript explicitly contained several of these elements. In contrast, two other human reviews and an exploratory source-grounded LLM review were more closely aligned with the manuscript.

The source-grounded LLM review provides an important nuance. Grounding improved the validity of the review-like output: it reduced severe false missing-content claims and focused criticism on genuine weaknesses. However, grounding did not eliminate hallucination. The output still contained a factual counting error and overstated a technical limitation.

Therefore, the lesson is not simply “AI cannot review papers.” The better conclusion is: ungrounded and unverified AI-assisted review is dangerous, while grounded AI assistance may be useful only when constrained, audited, disclosed where required, and ethically governed. No grounding tool can transfer responsibility away from the reviewer. A peer-review report is an official scholarly judgment, and the person whose name is attached to it must verify its claims.

Ethics and Confidentiality Statement

This paper analyzes anonymized peer-review materials from a real journal submission. The journal was a Korea Citation Index (KCI) candidate journal at the time of the review process. This broad indexing status is disclosed only to clarify that the case arose from a formal scholarly peer-review process. To reduce identifiability, the journal title, manuscript title, detailed application domain, system name, author identity in the original submission, reviewer identities, editorial metadata, submission dates, publication date, volume, issue, page information, and bibliographic links are omitted.

Selected technical metadata, including hardware, software frameworks, model configuration, model size, parameter count, and evaluation design, are retained because these details are necessary to analyze whether the review was grounded in the manuscript. The purpose of this paper is not to identify or accuse an individual reviewer, editor, or journal. The purpose is to analyze a failure mode in peer review and discuss safeguards for AI-assisted reviewing.

AI-Use Statement

AI tools were used to assist drafting, organization, and language editing of this manuscript. The human author remains solely responsible for the factual accuracy, interpretation, argumentation, and conclusions of the paper. All factual claims, citations, and case interpretations should be verified by the human author before public release.

References

[1] P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, et al., “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks,” Advances in Neural Information Processing Systems, 2020. URL: https://proceedings.neurips.cc/paper/2020/hash/6b493230205f780e1bc26945df7481e5-Abstract.html

[2] S. Borgeaud, A. Mensch, J. Hoffmann, T. Cai, E. Rutherford, K. Millican, et al., “Improving Language Models by Retrieving from Trillions of Tokens,” Proceedings of the 39th International Conference on Machine Learning, PMLR 162:2206–2240, 2022. URL: https://proceedings.mlr.press/v162/borgeaud22a.html

[3] A. Asai, Z. Wu, Y. Wang, A. Sil, and H. Hajishirzi, “Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection,” International Conference on Learning Representations, 2024. URL: https://arxiv.org/abs/2310.11511

[4] R. Nakano, J. Hilton, S. Balaji, J. Wu, L. Ouyang, C. Kim, et al., “WebGPT: Browser-assisted question-answering with human feedback,” arXiv preprint, 2021. URL: https://arxiv.org/abs/2112.09332

[5] W. H. Walters and E. I. Wilder, “Fabrication and errors in the bibliographic citations generated by ChatGPT,” Scientific Reports, 13, 14045, 2023. DOI: 10.1038/s41598-023-41032-5. URL: https://www.nature.com/articles/s41598-023-41032-5

[6] Nature Portfolio, “Artificial Intelligence (AI),” Editorial Policies. URL: https://www.nature.com/nature-portfolio/editorial-policies/ai. Accessed: May 4, 2026.

[7] SAGE, “Artificial Intelligence Policy,” Publication Ethics Policies. URL: https://www.sagepub.com/journals/publication-ethics-policies/artificial-intelligence-policy. Accessed: May 4, 2026.

[8] N. Lo Vecchio, “Personal experience with AI-generated peer reviews: a case study,” Research Integrity and Peer Review, 10, 4, 2025. DOI: 10.1186/s41073-025-00161-3. URL: https://link.springer.com/article/10.1186/s41073-025-00161-3

[9] Ban, Byunghyun, Donghun Ryu, and Minwoo Lee. "Machine learning approach to remove ion interference effect in agricultural nutrient solutions." 2019 International Conference on Information and Communication Technology Convergence (ICTC). IEEE, 2019.

[10] Ban, Byunghyun, et al. "Nutrient solution management system for smart farms and plant factory." 2020 international conference on information and communication technology convergence (ICTC). IEEE, 2020.

[11] Ban, Byunghyun, Minwoo Lee, and Donghun Ryu. "ODE network model for nonlinear and complex agricultural nutrient solution system." 2019 International Conference on Information and Communication Technology Convergence (ICTC). IEEE, 2019.

[12] Choo, Sang-Mok, et al. "The phenotype control kernel of a biomolecular regulatory network." BMC systems biology 12.1 (2018): 49.

[13] Ban, Byunghyun. "Deep learning method to remove chemical, kinetic and electric artifacts on ISEs." 2020 International Conference on Information and Communication Technology Convergence (ICTC). IEEE, 2020.

Appendix A. Anonymized Manuscript Evidence

Appendix B. Fair Rewriting of Review B Comments

Appendix C. Prompt Used for the Source-Grounded LLM Review

You are acting as a peer reviewer for a scientific manuscript.

Please write a peer review report for the uploaded manuscript. Evaluate the manuscript in terms of:

1. suitability of the research topic,

2. clarity of the abstract,

3. adequacy of the literature review,

4. methodological soundness,

5. experimental design and results,

6. appropriateness of references and citations,

7. originality,

8. academic contribution.

Please include:

- a brief summary of the manuscript,

- major strengths,

- major weaknesses,

- specific revision requests,

- an overall recommendation.

Use the style of a formal journal peer review report.