ATIQSER '26: "RefVerifier: Semi-automated ..."
RefVerifier: Semi-automated Reference Claim Verification for Scientific Manuscripts
Stefania Mocan, Florian Angermeir, and Mark Kreitz
(TU Munich, Germany; fortiss, Germany; Blekinge Institute of Technology, Sweden; University of the Bundeswehr Munich, Germany)
As software engineering research submission counts surge, peer reviewers face severe time constraints, making systematic verification of citation-supported claims prohibitively expensive. Consequently, unsubstantiated claims and semantic drift can propagate undetected across scientific literature. Existing approaches such as fact-checking and retrieval-augmented generation tools operate on open-domain web data or evaluate claims in isolation without processing complete manuscripts. To address this gap, we present RefVerifier, a semi-automated, citation-bounded reference verification prototype designed to support in academic peer review. RefVerifier extracts citation-bearing sentences from manuscripts, checks bibliography metadata against scholarly databases, resolves references to full-text open-access PDFs, localizes relevant evidence passages, and generates verdicts with natural language explanations. Evaluating RefVerifier on public benchmarks shows claim detection at an F1 score of 0.990, open-access resolution of 57.6% of references, and evidence localization with a hit rate of 98% on abstracts and 68% on complete cited papers. In an end-to-end test with eight manuscripts, RefVerifier achieves a verdict accuracy of 71%. By automating document retrieval and evidence localization while preserving reviewer oversight, RefVerifier provides first indicators for the feasibility of semi-automated integrity checks in scholarly publishing.
Article Search
Article: asews26atiqsermain-p3-p (type: Full Paper) doi:10.1145/3844135.3845866
ATIQSER '26: "doi-checker: A Tool for Validating ..."
doi-checker: A Tool for Validating References of Research Papers
Dirk Beyer and Marian Lingsch-Rosenfeld
(LMU Munich, Germany)
The bibliography of a research paper links its claims to prior work, but reference lists frequently contain digital object identifiers (DOIs) that are missing, invalid, or—most dangerously—resolvable yet pointing to a completely different publication. The last case is a threat to academic integrity and a hallmark of hallucinated ref- erences produced by AI-assisted writing. To improve upon this situation we present DOIchecker, a tool that extracts the refer- ences of a paper, resolves their metadata against the Crossref and DataCite registries, and reports for each reference whether its DOI is missing, invalid, or inconsistent with the cited work. In addi- tion, DOIchecker also checks the metadata of each reference for consistency with the listed authors. We evaluate DOIchecker on all the available open access proceedings of six major conferences (CAV, TACAS, POPL, PLDI, FSE, and ISSTA) between 2020 and 2026 covering the three page formats common in computer science: LNCS, ACM double column, and ACM single column. Across the 192 909 references of these 3 888 papers, the share that prints a DOI rises from 24.7 % in 2020 to 58.2 % in 2026. Of the DOIs that are printed, 0.82 % do not resolve, and for 6.43 % the registered meta- data disagrees with the cited entry. Setting aside a single volume whose references are extracted poorly, we find no evidence that such disagreements concentrate in recent years.
Article Search
Article: asews26atiqsermain-p1-p (type: Full Paper) doi:10.1145/3844135.3845864
ATIQSER '26: "ATIBA: Grounded Integrity ..."
ATIBA: Grounded Integrity and Quality Checking for Research Papers
Veli Karakaya, Semih Çağlar, Yusuf Yiğit Korkmaz, and Eray Tüzün
(Bilkent University, Türkiye)
Checking a manuscript's reference integrity, its compliance with a target venue's specific submission rules, and its adherence to community reporting standards is manual, repetitive, and different for every venue so in practice it is done inconsistently or skipped. We present ATIBA, a tool that runs five grounded integrity and quality checks on a manuscript: a reference-integrity check that verifies each citation against bibliographic sources and flags retracted or unfindable references; a venue/track compliance check that derives submission criteria directly from a venue's own call-for-papers page and evaluates the manuscript against them, each verdict anchored to a verbatim quote from that page; an empirical-standards compliance check against the ACM SIGSOFT Empirical Standards, with a hallucination defence that discards any evidence quote it cannot locate verbatim in the manuscript; a multi-mode AI review (venue-specific, formal, and page-anchored annotation) powered by GPT-5.4 through Azure OpenAI; and a citation-suggestion feature that proposes candidate references for a manuscript and verifies each against bibliographic sources before it is shown to the user. All five checks are designed around the same principle: an LLM is only trusted to judge, never to invent the evidence it judges against. We evaluated ATIBA through a moderated user study with 13 non-author participants. Agreement across the six survey items ranged from 69% to 92%, with a mean of 85%, providing initial evidence of positive perceived usefulness across the evaluated workflows. These findings establish perceived usefulness; objective accuracy remains to be measured.
Article Search
Article: asews26atiqsermain-p2-p (type: Full Paper) doi:10.1145/3844135.3845865
ATIQSER '26: "ATIBA: Grounded Integrity ..."
ATIBA: Grounded Integrity and Quality Checking for Research Papers
Veli Karakaya, Semih Çağlar, Yusuf Yiğit Korkmaz, and Eray Tüzün
(Bilkent University, Türkiye)
Checking a manuscript's reference integrity, its compliance with a target venue's specific submission rules, and its adherence to community reporting standards is manual, repetitive, and different for every venue so in practice it is done inconsistently or skipped. We present ATIBA, a tool that runs five grounded integrity and quality checks on a manuscript: a reference-integrity check that verifies each citation against bibliographic sources and flags retracted or unfindable references; a venue/track compliance check that derives submission criteria directly from a venue's own call-for-papers page and evaluates the manuscript against them, each verdict anchored to a verbatim quote from that page; an empirical-standards compliance check against the ACM SIGSOFT Empirical Standards, with a hallucination defence that discards any evidence quote it cannot locate verbatim in the manuscript; a multi-mode AI review (venue-specific, formal, and page-anchored annotation) powered by GPT-5.4 through Azure OpenAI; and a citation-suggestion feature that proposes candidate references for a manuscript and verifies each against bibliographic sources before it is shown to the user. All five checks are designed around the same principle: an LLM is only trusted to judge, never to invent the evidence it judges against. We evaluated ATIBA through a moderated user study with 13 non-author participants. Agreement across the six survey items ranged from 69% to 92%, with a mean of 85%, providing initial evidence of positive perceived usefulness across the evaluated workflows. These findings establish perceived usefulness; objective accuracy remains to be measured.
Article Search
Article: asews26atiqsermain-p2-p (type: Full Paper) doi:10.1145/3844135.3845865
ATIQSER '26: "ATIBA: Grounded Integrity ..."
ATIBA: Grounded Integrity and Quality Checking for Research Papers
Veli Karakaya, Semih Çağlar, Yusuf Yiğit Korkmaz, and Eray Tüzün
(Bilkent University, Türkiye)
Checking a manuscript's reference integrity, its compliance with a target venue's specific submission rules, and its adherence to community reporting standards is manual, repetitive, and different for every venue so in practice it is done inconsistently or skipped. We present ATIBA, a tool that runs five grounded integrity and quality checks on a manuscript: a reference-integrity check that verifies each citation against bibliographic sources and flags retracted or unfindable references; a venue/track compliance check that derives submission criteria directly from a venue's own call-for-papers page and evaluates the manuscript against them, each verdict anchored to a verbatim quote from that page; an empirical-standards compliance check against the ACM SIGSOFT Empirical Standards, with a hallucination defence that discards any evidence quote it cannot locate verbatim in the manuscript; a multi-mode AI review (venue-specific, formal, and page-anchored annotation) powered by GPT-5.4 through Azure OpenAI; and a citation-suggestion feature that proposes candidate references for a manuscript and verifies each against bibliographic sources before it is shown to the user. All five checks are designed around the same principle: an LLM is only trusted to judge, never to invent the evidence it judges against. We evaluated ATIBA through a moderated user study with 13 non-author participants. Agreement across the six survey items ranged from 69% to 92%, with a mean of 85%, providing initial evidence of positive perceived usefulness across the evaluated workflows. These findings establish perceived usefulness; objective accuracy remains to be measured.
Article Search
Article: asews26atiqsermain-p2-p (type: Full Paper) doi:10.1145/3844135.3845865
ATIQSER '26: "RefVerifier: Semi-automated ..."
RefVerifier: Semi-automated Reference Claim Verification for Scientific Manuscripts
Stefania Mocan, Florian Angermeir, and Mark Kreitz
(TU Munich, Germany; fortiss, Germany; Blekinge Institute of Technology, Sweden; University of the Bundeswehr Munich, Germany)
As software engineering research submission counts surge, peer reviewers face severe time constraints, making systematic verification of citation-supported claims prohibitively expensive. Consequently, unsubstantiated claims and semantic drift can propagate undetected across scientific literature. Existing approaches such as fact-checking and retrieval-augmented generation tools operate on open-domain web data or evaluate claims in isolation without processing complete manuscripts. To address this gap, we present RefVerifier, a semi-automated, citation-bounded reference verification prototype designed to support in academic peer review. RefVerifier extracts citation-bearing sentences from manuscripts, checks bibliography metadata against scholarly databases, resolves references to full-text open-access PDFs, localizes relevant evidence passages, and generates verdicts with natural language explanations. Evaluating RefVerifier on public benchmarks shows claim detection at an F1 score of 0.990, open-access resolution of 57.6% of references, and evidence localization with a hit rate of 98% on abstracts and 68% on complete cited papers. In an end-to-end test with eight manuscripts, RefVerifier achieves a verdict accuracy of 71%. By automating document retrieval and evidence localization while preserving reviewer oversight, RefVerifier provides first indicators for the feasibility of semi-automated integrity checks in scholarly publishing.
Article Search
Article: asews26atiqsermain-p3-p (type: Full Paper) doi:10.1145/3844135.3845866
ATIQSER '26: "doi-checker: A Tool for Validating ..."
doi-checker: A Tool for Validating References of Research Papers
Dirk Beyer and Marian Lingsch-Rosenfeld
(LMU Munich, Germany)
The bibliography of a research paper links its claims to prior work, but reference lists frequently contain digital object identifiers (DOIs) that are missing, invalid, or—most dangerously—resolvable yet pointing to a completely different publication. The last case is a threat to academic integrity and a hallmark of hallucinated ref- erences produced by AI-assisted writing. To improve upon this situation we present DOIchecker, a tool that extracts the refer- ences of a paper, resolves their metadata against the Crossref and DataCite registries, and reports for each reference whether its DOI is missing, invalid, or inconsistent with the cited work. In addi- tion, DOIchecker also checks the metadata of each reference for consistency with the listed authors. We evaluate DOIchecker on all the available open access proceedings of six major conferences (CAV, TACAS, POPL, PLDI, FSE, and ISSTA) between 2020 and 2026 covering the three page formats common in computer science: LNCS, ACM double column, and ACM single column. Across the 192 909 references of these 3 888 papers, the share that prints a DOI rises from 24.7 % in 2020 to 58.2 % in 2026. Of the DOIs that are printed, 0.82 % do not resolve, and for 6.43 % the registered meta- data disagrees with the cited entry. Setting aside a single volume whose references are extracted poorly, we find no evidence that such disagreements concentrate in recent years.
Article Search
Article: asews26atiqsermain-p1-p (type: Full Paper) doi:10.1145/3844135.3845864
ATIQSER '26: "RefVerifier: Semi-automated ..."
RefVerifier: Semi-automated Reference Claim Verification for Scientific Manuscripts
Stefania Mocan, Florian Angermeir, and Mark Kreitz
(TU Munich, Germany; fortiss, Germany; Blekinge Institute of Technology, Sweden; University of the Bundeswehr Munich, Germany)
As software engineering research submission counts surge, peer reviewers face severe time constraints, making systematic verification of citation-supported claims prohibitively expensive. Consequently, unsubstantiated claims and semantic drift can propagate undetected across scientific literature. Existing approaches such as fact-checking and retrieval-augmented generation tools operate on open-domain web data or evaluate claims in isolation without processing complete manuscripts. To address this gap, we present RefVerifier, a semi-automated, citation-bounded reference verification prototype designed to support in academic peer review. RefVerifier extracts citation-bearing sentences from manuscripts, checks bibliography metadata against scholarly databases, resolves references to full-text open-access PDFs, localizes relevant evidence passages, and generates verdicts with natural language explanations. Evaluating RefVerifier on public benchmarks shows claim detection at an F1 score of 0.990, open-access resolution of 57.6% of references, and evidence localization with a hit rate of 98% on abstracts and 68% on complete cited papers. In an end-to-end test with eight manuscripts, RefVerifier achieves a verdict accuracy of 71%. By automating document retrieval and evidence localization while preserving reviewer oversight, RefVerifier provides first indicators for the feasibility of semi-automated integrity checks in scholarly publishing.
Article Search
Article: asews26atiqsermain-p3-p (type: Full Paper) doi:10.1145/3844135.3845866
ATIQSER '26: "ATIBA: Grounded Integrity ..."
ATIBA: Grounded Integrity and Quality Checking for Research Papers
Veli Karakaya, Semih Çağlar, Yusuf Yiğit Korkmaz, and Eray Tüzün
(Bilkent University, Türkiye)
Checking a manuscript's reference integrity, its compliance with a target venue's specific submission rules, and its adherence to community reporting standards is manual, repetitive, and different for every venue so in practice it is done inconsistently or skipped. We present ATIBA, a tool that runs five grounded integrity and quality checks on a manuscript: a reference-integrity check that verifies each citation against bibliographic sources and flags retracted or unfindable references; a venue/track compliance check that derives submission criteria directly from a venue's own call-for-papers page and evaluates the manuscript against them, each verdict anchored to a verbatim quote from that page; an empirical-standards compliance check against the ACM SIGSOFT Empirical Standards, with a hallucination defence that discards any evidence quote it cannot locate verbatim in the manuscript; a multi-mode AI review (venue-specific, formal, and page-anchored annotation) powered by GPT-5.4 through Azure OpenAI; and a citation-suggestion feature that proposes candidate references for a manuscript and verifies each against bibliographic sources before it is shown to the user. All five checks are designed around the same principle: an LLM is only trusted to judge, never to invent the evidence it judges against. We evaluated ATIBA through a moderated user study with 13 non-author participants. Agreement across the six survey items ranged from 69% to 92%, with a mean of 85%, providing initial evidence of positive perceived usefulness across the evaluated workflows. These findings establish perceived usefulness; objective accuracy remains to be measured.
Article Search
Article: asews26atiqsermain-p2-p (type: Full Paper) doi:10.1145/3844135.3845865