Workshop ATIQSER 2026 – Author Index |
Contents -
Abstracts -
Authors
|
| Angermeir, Florian |
Stefania Mocan, Florian Angermeir, and Mark Kreitz (TU Munich, Germany; fortiss, Germany; Blekinge Institute of Technology, Sweden; University of the Bundeswehr Munich, Germany) As software engineering research submission counts surge, peer reviewers face severe time constraints, making systematic verification of citation-supported claims prohibitively expensive. Consequently, unsubstantiated claims and semantic drift can propagate undetected across scientific literature. Existing approaches such as fact-checking and retrieval-augmented generation tools operate on open-domain web data or evaluate claims in isolation without processing complete manuscripts. To address this gap, we present RefVerifier, a semi-automated, citation-bounded reference verification prototype designed to support in academic peer review. RefVerifier extracts citation-bearing sentences from manuscripts, checks bibliography metadata against scholarly databases, resolves references to full-text open-access PDFs, localizes relevant evidence passages, and generates verdicts with natural language explanations. Evaluating RefVerifier on public benchmarks shows claim detection at an F1 score of 0.990, open-access resolution of 57.6% of references, and evidence localization with a hit rate of 98% on abstracts and 68% on complete cited papers. In an end-to-end test with eight manuscripts, RefVerifier achieves a verdict accuracy of 71%. By automating document retrieval and evidence localization while preserving reviewer oversight, RefVerifier provides first indicators for the feasibility of semi-automated integrity checks in scholarly publishing. |
|
| Beyer, Dirk |
Dirk Beyer and Marian Lingsch-Rosenfeld (LMU Munich, Germany) The reference list of a research paper links its claims to prior work. However, they frequently contain references without digital object identifiers (DOIs), invalid DOIs, or valid DOIs pointing to a different publication. The last case is a threat to academic integrity. To improve upon this situation, we present doi-checker, a tool that extracts the references of a paper, resolves their metadata against the Crossref and DataCite registries, and reports for each reference whether its DOI is missing, invalid, or inconsistent with the cited work. In addition, doi-checker also checks the metadata of each reference for consistency with the listed authors. We evaluate doi-checker on all the available open-access proceedings of six major conferences (CAV, TACAS, POPL, PLDI, FSE, and ISSTA) from 2020 to 2026, covering three page formats common in computer science: LNCS, ACM double column, and ACM single column. Across the 193 770 references of these 3 888 papers, the share for which doi-checker finds DOIs rises from 26.0 % in 2020 to 60.7 % in 2026. Of these, 0.47 % do not resolve, and for 5.79 %, the registered metadata disagrees with the cited entry. Setting aside a single volume for which reference extraction is poor, we find no evidence that such disagreements concentrate in recent years. |
|
| Çağlar, Semih |
Veli Karakaya, Semih Çağlar, Yusuf Yiğit Korkmaz, and Eray Tüzün (Bilkent University, Türkiye) Checking a manuscript's reference integrity, its compliance with a target venue's specific submission rules, and its adherence to community reporting standards is manual, repetitive, and different for every venue so in practice it is done inconsistently or skipped. We present ATIBA, a tool that runs five grounded integrity and quality checks on a manuscript: a reference-integrity check that verifies each citation against bibliographic sources and flags retracted or unfindable references; a venue/track compliance check that derives submission criteria directly from a venue's own call-for-papers page and evaluates the manuscript against them, each verdict anchored to a verbatim quote from that page; an empirical-standards compliance check against the ACM SIGSOFT Empirical Standards, with a hallucination defence that discards any evidence quote it cannot locate verbatim in the manuscript; a multi-mode AI review (venue-specific, formal, and page-anchored annotation) powered by GPT-5.4 through Azure OpenAI; and a citation-suggestion feature that proposes candidate references for a manuscript and verifies each against bibliographic sources before it is shown to the user. All five checks are designed around the same principle: an LLM is only trusted to judge, never to invent the evidence it judges against. We evaluated ATIBA through a moderated user study with 13 non-author participants. Agreement across the six survey items ranged from 69% to 92%, with a mean of 85%, providing initial evidence of positive perceived usefulness across the evaluated workflows. These findings establish perceived usefulness; objective accuracy remains to be measured. |
|
| Karakaya, Veli |
Veli Karakaya, Semih Çağlar, Yusuf Yiğit Korkmaz, and Eray Tüzün (Bilkent University, Türkiye) Checking a manuscript's reference integrity, its compliance with a target venue's specific submission rules, and its adherence to community reporting standards is manual, repetitive, and different for every venue so in practice it is done inconsistently or skipped. We present ATIBA, a tool that runs five grounded integrity and quality checks on a manuscript: a reference-integrity check that verifies each citation against bibliographic sources and flags retracted or unfindable references; a venue/track compliance check that derives submission criteria directly from a venue's own call-for-papers page and evaluates the manuscript against them, each verdict anchored to a verbatim quote from that page; an empirical-standards compliance check against the ACM SIGSOFT Empirical Standards, with a hallucination defence that discards any evidence quote it cannot locate verbatim in the manuscript; a multi-mode AI review (venue-specific, formal, and page-anchored annotation) powered by GPT-5.4 through Azure OpenAI; and a citation-suggestion feature that proposes candidate references for a manuscript and verifies each against bibliographic sources before it is shown to the user. All five checks are designed around the same principle: an LLM is only trusted to judge, never to invent the evidence it judges against. We evaluated ATIBA through a moderated user study with 13 non-author participants. Agreement across the six survey items ranged from 69% to 92%, with a mean of 85%, providing initial evidence of positive perceived usefulness across the evaluated workflows. These findings establish perceived usefulness; objective accuracy remains to be measured. |
|
| Korkmaz, Yusuf Yiğit |
Veli Karakaya, Semih Çağlar, Yusuf Yiğit Korkmaz, and Eray Tüzün (Bilkent University, Türkiye) Checking a manuscript's reference integrity, its compliance with a target venue's specific submission rules, and its adherence to community reporting standards is manual, repetitive, and different for every venue so in practice it is done inconsistently or skipped. We present ATIBA, a tool that runs five grounded integrity and quality checks on a manuscript: a reference-integrity check that verifies each citation against bibliographic sources and flags retracted or unfindable references; a venue/track compliance check that derives submission criteria directly from a venue's own call-for-papers page and evaluates the manuscript against them, each verdict anchored to a verbatim quote from that page; an empirical-standards compliance check against the ACM SIGSOFT Empirical Standards, with a hallucination defence that discards any evidence quote it cannot locate verbatim in the manuscript; a multi-mode AI review (venue-specific, formal, and page-anchored annotation) powered by GPT-5.4 through Azure OpenAI; and a citation-suggestion feature that proposes candidate references for a manuscript and verifies each against bibliographic sources before it is shown to the user. All five checks are designed around the same principle: an LLM is only trusted to judge, never to invent the evidence it judges against. We evaluated ATIBA through a moderated user study with 13 non-author participants. Agreement across the six survey items ranged from 69% to 92%, with a mean of 85%, providing initial evidence of positive perceived usefulness across the evaluated workflows. These findings establish perceived usefulness; objective accuracy remains to be measured. |
|
| Kreitz, Mark |
Stefania Mocan, Florian Angermeir, and Mark Kreitz (TU Munich, Germany; fortiss, Germany; Blekinge Institute of Technology, Sweden; University of the Bundeswehr Munich, Germany) As software engineering research submission counts surge, peer reviewers face severe time constraints, making systematic verification of citation-supported claims prohibitively expensive. Consequently, unsubstantiated claims and semantic drift can propagate undetected across scientific literature. Existing approaches such as fact-checking and retrieval-augmented generation tools operate on open-domain web data or evaluate claims in isolation without processing complete manuscripts. To address this gap, we present RefVerifier, a semi-automated, citation-bounded reference verification prototype designed to support in academic peer review. RefVerifier extracts citation-bearing sentences from manuscripts, checks bibliography metadata against scholarly databases, resolves references to full-text open-access PDFs, localizes relevant evidence passages, and generates verdicts with natural language explanations. Evaluating RefVerifier on public benchmarks shows claim detection at an F1 score of 0.990, open-access resolution of 57.6% of references, and evidence localization with a hit rate of 98% on abstracts and 68% on complete cited papers. In an end-to-end test with eight manuscripts, RefVerifier achieves a verdict accuracy of 71%. By automating document retrieval and evidence localization while preserving reviewer oversight, RefVerifier provides first indicators for the feasibility of semi-automated integrity checks in scholarly publishing. |
|
| Lingsch-Rosenfeld, Marian |
Dirk Beyer and Marian Lingsch-Rosenfeld (LMU Munich, Germany) The reference list of a research paper links its claims to prior work. However, they frequently contain references without digital object identifiers (DOIs), invalid DOIs, or valid DOIs pointing to a different publication. The last case is a threat to academic integrity. To improve upon this situation, we present doi-checker, a tool that extracts the references of a paper, resolves their metadata against the Crossref and DataCite registries, and reports for each reference whether its DOI is missing, invalid, or inconsistent with the cited work. In addition, doi-checker also checks the metadata of each reference for consistency with the listed authors. We evaluate doi-checker on all the available open-access proceedings of six major conferences (CAV, TACAS, POPL, PLDI, FSE, and ISSTA) from 2020 to 2026, covering three page formats common in computer science: LNCS, ACM double column, and ACM single column. Across the 193 770 references of these 3 888 papers, the share for which doi-checker finds DOIs rises from 26.0 % in 2020 to 60.7 % in 2026. Of these, 0.47 % do not resolve, and for 5.79 %, the registered metadata disagrees with the cited entry. Setting aside a single volume for which reference extraction is poor, we find no evidence that such disagreements concentrate in recent years. |
|
| Mocan, Stefania |
Stefania Mocan, Florian Angermeir, and Mark Kreitz (TU Munich, Germany; fortiss, Germany; Blekinge Institute of Technology, Sweden; University of the Bundeswehr Munich, Germany) As software engineering research submission counts surge, peer reviewers face severe time constraints, making systematic verification of citation-supported claims prohibitively expensive. Consequently, unsubstantiated claims and semantic drift can propagate undetected across scientific literature. Existing approaches such as fact-checking and retrieval-augmented generation tools operate on open-domain web data or evaluate claims in isolation without processing complete manuscripts. To address this gap, we present RefVerifier, a semi-automated, citation-bounded reference verification prototype designed to support in academic peer review. RefVerifier extracts citation-bearing sentences from manuscripts, checks bibliography metadata against scholarly databases, resolves references to full-text open-access PDFs, localizes relevant evidence passages, and generates verdicts with natural language explanations. Evaluating RefVerifier on public benchmarks shows claim detection at an F1 score of 0.990, open-access resolution of 57.6% of references, and evidence localization with a hit rate of 98% on abstracts and 68% on complete cited papers. In an end-to-end test with eight manuscripts, RefVerifier achieves a verdict accuracy of 71%. By automating document retrieval and evidence localization while preserving reviewer oversight, RefVerifier provides first indicators for the feasibility of semi-automated integrity checks in scholarly publishing. |
|
| Tüzün, Eray |
Veli Karakaya, Semih Çağlar, Yusuf Yiğit Korkmaz, and Eray Tüzün (Bilkent University, Türkiye) Checking a manuscript's reference integrity, its compliance with a target venue's specific submission rules, and its adherence to community reporting standards is manual, repetitive, and different for every venue so in practice it is done inconsistently or skipped. We present ATIBA, a tool that runs five grounded integrity and quality checks on a manuscript: a reference-integrity check that verifies each citation against bibliographic sources and flags retracted or unfindable references; a venue/track compliance check that derives submission criteria directly from a venue's own call-for-papers page and evaluates the manuscript against them, each verdict anchored to a verbatim quote from that page; an empirical-standards compliance check against the ACM SIGSOFT Empirical Standards, with a hallucination defence that discards any evidence quote it cannot locate verbatim in the manuscript; a multi-mode AI review (venue-specific, formal, and page-anchored annotation) powered by GPT-5.4 through Azure OpenAI; and a citation-suggestion feature that proposes candidate references for a manuscript and verifies each against bibliographic sources before it is shown to the user. All five checks are designed around the same principle: an LLM is only trusted to judge, never to invent the evidence it judges against. We evaluated ATIBA through a moderated user study with 13 non-author participants. Agreement across the six survey items ranged from 69% to 92%, with a mean of 85%, providing initial evidence of positive perceived usefulness across the evaluated workflows. These findings establish perceived usefulness; objective accuracy remains to be measured. |
9 authors
proc time: 0.04