POVC '26: "VibeCheck: Assessing the Quality ..."
VibeCheck: Assessing the Quality of LLM-Generated Unit Tests: A Multi-agent Empirical Study across Heterogeneous Repositories
Anika Tabassum, Mushahid Intesum, Md. Fahim Arefin, and Tarannum Shaila Zaman
(University of Dhaka, Bangladesh; University of Maryland, Baltimore County, USA)
LLM-based IDE agents are increasingly used to generate repository-grounded unit tests, yet common evaluations often rely on execution success or coverage. These metrics can miss deeper quality issues such as weak assertions, missing edge cases, poor isolation, and limited maintainability. This paper presents VibeCheck, an empirical study of unit test generation across 15 student-developed Python and JavaScript/TypeScript repositories. We evaluate Kiro, Antigravity, and Cursor with Claude Sonnet 4.5 as the underlying agent, under repository-only, zero-shot conditions using a five-dimensional rubric covering runnability, assertion strength, logic and edge-case coverage, isolation/determinism, and maintainability. We also apply leave-one-out cross-agent peer evaluation to compare tools and identify failure patterns. Results show a clear execution-adequacy gap: generated tests are often runnable but frequently lack strong assertions and meaningful behavioral coverage. Weak assertions and missing edge cases occur more often than blocking failures, showing that runnable tests can still be shallow. VibeCheck provides a reliability-oriented framework for evaluating LLM-generated tests beyond pass/fail outcomes.
Article Search
Article: asews26povcmain-p7-p (type: Full Paper) doi:10.1145/3843779.3844632
POVC '26: "Prompts Matter: On the Impact ..."
Prompts Matter: On the Impact of Code Smell Remediation with Large Language Models
Nadim Hagatulah, Markus Borg, and Emma Söderberg
(Lund University, Sweden; CodeScene, Sweden)
Code smells accumulate as technical debt, reducing software maintainability. Refactoring is the established remediation practice, but it is complex and often deprioritized. Large Language Models can support automated refactoring, but their effectiveness depends on how they are prompted. In this study, we evaluate 14 prompt configurations with GPT-OSS and Qwen3 on 2,000 Java and Python functions, producing 168,000 function-level refactorings. We evaluate behavior preservation, code smell remediation, and cost by token proxy. Results show that prompt effectiveness is contextual. For GPT-OSS, prompts combining explicit smell thresholds with reasoning triggers increased remediation odds by up to 1.32× over zero-shot at a 1.05–1.10× cost multiplier. For Qwen3, prompts eliciting reasoning reduced behavior-preservation odds to 0.75× of zero-shot and raised cost by up to 1.51× zero-shot. Conversely, the threshold-only prompt increased remediation odds by up to 1.18× with near-neutral overhead (1.01–1.02×). Cost per successful refactoring was 1.01–1.04× ZS for GPT-OSS and 1.12–1.21× for Qwen3. Overall, prompt choice matters, but model choice produced the larger difference. These findings support validation-centered, model-specific prompt selection rather than a uniform strategy.
Article Search
Article: asews26povcmain-p18-p (type: Full Paper) doi:10.1145/3843779.3844637
POVC '26: "Geometry-Aware Prompt Context ..."
Geometry-Aware Prompt Context for Model Consistency Management
Monalisha Ojha, Shilpi Gupta, and Rahul Sharma
(University of Mannheim, Germany; Tallinn University of Technology, Estonia)
Keeping independently developed models consistent with a large language model means prompting it with two metamodels at once, yet industrial metamodels hold hundreds of classifiers and cannot be pasted in. The usual remedy---serialise, chunk, and retrieve by text or embedding similarity---flattens inheritance, containment and typed references into linear text, so retrieval compares names rather than structure, and fails when two teams name the same concept differently. A metamodel already carries an exact metric: its hierarchy induces a geodesic distance that needs no training and that an embedding can only approximate. That metric holds within one metamodel, not across two; what joins them is a few correspondences confirmed by an engineer. We present Anchored Geometric Propagation (AGP): it joins two metamodels at these anchors, ranks candidates by geodesic distance blended with name similarity at a weight calibrated on the anchors, promotes confident matches to new anchors, and defers the rest to the engineer. Selected classifiers carry their supertypes and feature types, so the prompt holds a well-formed fragment of the hierarchy. On metamodel pairs derived from ModelSet, AGP covers the large majority of queries at a fraction of the cost of full serialisation, and outperforms lexical retrieval precisely where lexical retrieval fails. The evaluation is intrinsic: it measures what reaches the prompt, not what a language model subsequently does with it, and so establishes a necessary condition for the downstream task rather than an accuracy gain.
Article Search
Article: asews26povcmain-p20-p (type: Full Paper) doi:10.1145/3843779.3844638
POVC '26: "Prompts Matter: On the Impact ..."
Prompts Matter: On the Impact of Code Smell Remediation with Large Language Models
Nadim Hagatulah, Markus Borg, and Emma Söderberg
(Lund University, Sweden; CodeScene, Sweden)
Code smells accumulate as technical debt, reducing software maintainability. Refactoring is the established remediation practice, but it is complex and often deprioritized. Large Language Models can support automated refactoring, but their effectiveness depends on how they are prompted. In this study, we evaluate 14 prompt configurations with GPT-OSS and Qwen3 on 2,000 Java and Python functions, producing 168,000 function-level refactorings. We evaluate behavior preservation, code smell remediation, and cost by token proxy. Results show that prompt effectiveness is contextual. For GPT-OSS, prompts combining explicit smell thresholds with reasoning triggers increased remediation odds by up to 1.32× over zero-shot at a 1.05–1.10× cost multiplier. For Qwen3, prompts eliciting reasoning reduced behavior-preservation odds to 0.75× of zero-shot and raised cost by up to 1.51× zero-shot. Conversely, the threshold-only prompt increased remediation odds by up to 1.18× with near-neutral overhead (1.01–1.02×). Cost per successful refactoring was 1.01–1.04× ZS for GPT-OSS and 1.12–1.21× for Qwen3. Overall, prompt choice matters, but model choice produced the larger difference. These findings support validation-centered, model-specific prompt selection rather than a uniform strategy.
Article Search
Article: asews26povcmain-p18-p (type: Full Paper) doi:10.1145/3843779.3844637
POVC '26: "Toward Trace-Guided Automated ..."
Toward Trace-Guided Automated Program Repair
David Georg Reichelt, Shinhyung Yang, and Wilhelm Hasselbring
(Lancaster University Leipzig, Germany; Universität Leipzig, Germany; Kiel University, Germany)
Fixing bugs is a time-consuming task. Researchers and developers hence aim to automate program repair (APR) as much as possible. Recent advances in Large Language Models (LLMs) made automated fixing of bugs widely accessible to practitioners. However, while LLMs are in principle capable of generating code and fixing programs, their ability to locate and therefore effectively fix bugs is limited by their context length. To guide them towards the parts of the program that require fixing, LLMs could be prompted with information from dynamic analysis techniques.
We expect that the most efficient way to help LLMs debug code is by providing the LLM with information to interactively find the bug location. To do so, we propose two steps: Tailoring the prompt with trace information and giving the LLM a tool call to get further trace information. This trace information needs to be preprocessed in a way that it can be efficiently used by the LLM and makes it possible to understand the program behavior.
In this paper, we describe our concept of an APR approach that is guided by tracing data and which is therefore capable of fixing bugs in huge codebases. As a proof of concept, we show that in the bug dataset defects4j, location information achieves an absolute increase of up to 22.4 % in fixed bugs with Gemma-4-31B compared to a baseline without location information. Based on our concept and our proof of concept, we outline the research agenda that is necessary to enable trace-guided APR.
Article Search
Article: asews26povcmain-p4-p (type: Full Paper) doi:10.1145/3843779.3844631
POVC '26: "VibeCheck: Assessing the Quality ..."
VibeCheck: Assessing the Quality of LLM-Generated Unit Tests: A Multi-agent Empirical Study across Heterogeneous Repositories
Anika Tabassum, Mushahid Intesum, Md. Fahim Arefin, and Tarannum Shaila Zaman
(University of Dhaka, Bangladesh; University of Maryland, Baltimore County, USA)
LLM-based IDE agents are increasingly used to generate repository-grounded unit tests, yet common evaluations often rely on execution success or coverage. These metrics can miss deeper quality issues such as weak assertions, missing edge cases, poor isolation, and limited maintainability. This paper presents VibeCheck, an empirical study of unit test generation across 15 student-developed Python and JavaScript/TypeScript repositories. We evaluate Kiro, Antigravity, and Cursor with Claude Sonnet 4.5 as the underlying agent, under repository-only, zero-shot conditions using a five-dimensional rubric covering runnability, assertion strength, logic and edge-case coverage, isolation/determinism, and maintainability. We also apply leave-one-out cross-agent peer evaluation to compare tools and identify failure patterns. Results show a clear execution-adequacy gap: generated tests are often runnable but frequently lack strong assertions and meaningful behavioral coverage. Weak assertions and missing edge cases occur more often than blocking failures, showing that runnable tests can still be shallow. VibeCheck provides a reliability-oriented framework for evaluating LLM-generated tests beyond pass/fail outcomes.
Article Search
Article: asews26povcmain-p7-p (type: Full Paper) doi:10.1145/3843779.3844632
POVC '26: "Your Prompt Is Only Half the ..."
Your Prompt Is Only Half the Prompt: Silent Chat-Template Drift in Open-Weight Model Repositories
Simarjot Khanna
(Independent, Canada)
PromptOps treats prompts as code: versioned, reviewed, and defended by a regression suite. The prompt a model actually reads is not the prompt a team writes. It is first rewritten by the model's chat template, an executable Jinja program shipped inside the model repository, edited in place, with no version of its own. How often this hidden dependency changes, and with what behavioural consequence, has not been measured. We reconstruct the default-branch template history of 400 of the most-downloaded text-generation repositories on the Hugging Face Hub. A renderer validated byte-for-byte against the reference implementation decides whether each edit changes what the model is shown, and we run 5 local checkpoints under their own repositories' historical templates. 43.8% of the repositories changed their template after first publishing one, and 80.6% of the change events moved no weight file, invisible to a team that pinned the model and verified its weights; 50.2% of the decidable edits change what the model is shown. Much of the raw rate is launch-week churn, and we report the settled figure beside it: 8.5% of the frame saw such an edit land more than 30 days after its first template. In a one-repository case study, an edit that dropped the system role halved output-contract compliance from 90.0% to 45.0% until a later edit largely restored it. templatelock pins what a template renders rather than what it says, separating the edits that matter from the ones that do not at 9.2 ms per model. Until prompt pipelines version the template, pinning the prompt and the model does not pin the model's input.
Article Search
Article: asews26povcmain-p16-p (type: Full Paper) doi:10.1145/3843779.3844636
POVC '26: "TurnCost: Turns, Not Observations: ..."
TurnCost: Turns, Not Observations: Cache-Aware Loop Engineering for Coding Agents
Pramod Kumar and Raj Kumar Pandey
(Pennsylvania State University, University Park, USA)
AI coding agents operate in iterative loops: they modify code, run tools, inspect the results, and decide what to do next. We call the design of this process loop engineering, which includes context management, model selection, compaction, and termination. Existing efficiency work mainly compresses tool observations or routes selected turns to cheaper models, but the practical opportunity for these optimizations remains unclear. We analyze 600 publicly released coding-agent trajectories spanning two agent scaffolds and two language models. Median observations contain only 118-286 tokens, while the large build logs commonly used to motivate compression occur near the ninety-ninth percentile. Moreover, 98.6% of input tokens are served from a prefix cache, so flat input pricing overestimates loop cost by roughly 7x on average under near-ideal caching.
We derive and empirically validate a closed-form break-even condition for model delegation. Delegation increases cost in half of the evaluated settings, near the predicted boundary. More broadly, turn count explains 84% of the variance in log trajectory cost, compared with 4% for mean observation size; the most expensive trajectories have the most turns despite having the smallest observations. These associations suggest that observation-focused methods target a secondary cost factor, and identify unnecessary turns as the larger and largely unexamined opportunity, though we do not establish that turns can be removed while holding task success fixed. We release TurnCost, a tool for re-scoring trajectory corpora under alternative caching, pricing, compression, and routing policies.
Article Search
Artifacts Available
Article: asews26povcmain-p15-p (type: Full Paper) doi:10.1145/3843779.3844635
POVC '26: "Agent4RE: A Self-Refining ..."
Agent4RE: A Self-Refining Multi-agent Framework for End-to-End Software Requirements Engineering and Benchmarking
Yongjian Tang, Linhan Li, and Thomas Runkler
(Siemens, Germany; TU Munich, Germany)
Existing LLM-based approaches for software Requirements Engineering (RE) typically rely on basic prompting strategies or rudimentary agent collaboration, under-utilizing the full potential of multi-agent systems. Meanwhile, available datasets focus on isolated subtasks, such as requirements extraction, classification, and completeness detection, leaving the absence of an end-to-end RE benchmark that spans from requirements elicitation to generation. We present Agent4RE - a self-refining multi-agent RE system that orchestrates specialized agents and incorporates two iterative improvement loops. To support evaluation, we construct RE-E2E - a real-world dataset built from human-written requirement specifications, enabling end-to-end assessment of RE workflows. Building on this foundation, we further propose two enhanced Agent4RE versions that incorporate either autonomous self-refinement or structured human feedback, and analyze their strengths and limitations across different scenarios. Evaluation on 8 Large Language Models (LLMs) demonstrates that all three Agent4RE variants consistently outperform a domain-context-augmented prompting baseline by average 8% in text-based metrics. The two enhanced variants achieve the highest LLM-as-a-judge and human ratings, surpassing two RE baselines by approximately 0.8 points on a four-point scale. This consistent performance establishes Agent4RE as a practical end-to-end RE solution for industrial environments.
Article Search
Article: asews26povcmain-p11-p (type: Full Paper) doi:10.1145/3843779.3844634
POVC '26: "Ranking Is Not Authority: ..."
Ranking Is Not Authority: Executable Context-Routing Contracts for Coding Agents
Nikhil Mahesh
(Independent, USA)
Coding agents rely on context routers to choose repository evidence before generation. A learned relevance score can be used for two different decisions: ranking evidence and deleting it from the initial payload. We introduce an executable routing contract that separates those powers. Learned scores may reorder a deterministic lexical baseline, but only deterministic policy may delete it; a pre-generation digest verifies that the implementation obeys this by-construction rule. We evaluate five routing policies in 200 assignments on 50 frozen SWE-bench Verified tasks and analyze the disjoint 450-task complement. The advisory implementation retained all 1,638 admitted spans without a digest violation. Of 6,000 candidate scores, 2,840 were operationally unavailable; 91.3% of those unavailable scores ended in recorded account-credit or routing/fallback records. At threshold 6, destructive reuse would delete every admitted correct-edit span on 19 of 31 reachable tasks, with 18 of the 19 deletions caused by treating unavailable scores as removal rather than by a low numeric score. In the actual destructive arm, five of seven tasks still resolved after complete initial edit-locus deletion because repository tools allowed later recovery. We contribute the executable contract and its conformance tests, an attempt-complete evaluation ledger, and evidence that aggregate repair success cannot certify what an agent received at the initial-context boundary.
Article Search
Artifacts Available
Article: asews26povcmain-p21-p (type: Full Paper) doi:10.1145/3843779.3844639
POVC '26: "Prompt Pipelines as Engineered ..."
Prompt Pipelines as Engineered Artifacts: Regression-Testing a Multi-agent Threat-Modeling Pipeline
Eldar Mametov, Andrey Sadovykh, Eugene Zouev, and Alberto Sillitti
(Innopolis University, Russia; SOFTEAM, France; Centre for Applied Software Engineering, Italy)
Security pipelines built on large language models (LLMs) are, at their core, collections of prompts: role instructions, retrieval queries, critique templates, and output formats. In practice these prompts evolve the way early software did: by trial and error, with changes that are rarely documented and never regression-tested. This paper reports engineering experience from treating a multi-agent threat-modeling pipeline as a set of versioned, tested software artifacts. The pipeline decomposes STRIDE analysis into six specialised agent prompts with retrieval enrichment and a critique stage; a fixed corpus of 24 system architectures with expert-curated reference threats serves as a regression oracle for every change to a prompt or pipeline stage. We describe three documented design decisions the harness drove: removing a chunk-filtered retrieval stage that consumed tokens without improving coverage, declining to adopt the most expensive pipeline as the default after the oracle showed a six-agent decomposition matched its coverage at roughly one-ninth the token cost, and catching a metric-specification drift between documentation and implementation. We distil the recurring failure modes (retrieval-induced specificity bias, mechanism-versus-impact category confusion, and systematic risk over-estimation) into checks that now guard the pipeline. The experience suggests that the PromptOps practices recently proposed for general LLM applications transfer directly to security engineering, and that a fixed evaluation corpus with cost-aware metrics is the single most valuable artifact a prompt-pipeline team can maintain.
Article Search
Artifacts Available
Article: asews26povcmain-p10-p (type: Full Paper) doi:10.1145/3843779.3844633
POVC '26: "Geometry-Aware Prompt Context ..."
Geometry-Aware Prompt Context for Model Consistency Management
Monalisha Ojha, Shilpi Gupta, and Rahul Sharma
(University of Mannheim, Germany; Tallinn University of Technology, Estonia)
Keeping independently developed models consistent with a large language model means prompting it with two metamodels at once, yet industrial metamodels hold hundreds of classifiers and cannot be pasted in. The usual remedy---serialise, chunk, and retrieve by text or embedding similarity---flattens inheritance, containment and typed references into linear text, so retrieval compares names rather than structure, and fails when two teams name the same concept differently. A metamodel already carries an exact metric: its hierarchy induces a geodesic distance that needs no training and that an embedding can only approximate. That metric holds within one metamodel, not across two; what joins them is a few correspondences confirmed by an engineer. We present Anchored Geometric Propagation (AGP): it joins two metamodels at these anchors, ranks candidates by geodesic distance blended with name similarity at a weight calibrated on the anchors, promotes confident matches to new anchors, and defers the rest to the engineer. Selected classifiers carry their supertypes and feature types, so the prompt holds a well-formed fragment of the hierarchy. On metamodel pairs derived from ModelSet, AGP covers the large majority of queries at a fraction of the cost of full serialisation, and outperforms lexical retrieval precisely where lexical retrieval fails. The evaluation is intrinsic: it measures what reaches the prompt, not what a language model subsequently does with it, and so establishes a necessary condition for the downstream task rather than an accuracy gain.
Article Search
Article: asews26povcmain-p20-p (type: Full Paper) doi:10.1145/3843779.3844638
POVC '26: "TurnCost: Turns, Not Observations: ..."
TurnCost: Turns, Not Observations: Cache-Aware Loop Engineering for Coding Agents
Pramod Kumar and Raj Kumar Pandey
(Pennsylvania State University, University Park, USA)
AI coding agents operate in iterative loops: they modify code, run tools, inspect the results, and decide what to do next. We call the design of this process loop engineering, which includes context management, model selection, compaction, and termination. Existing efficiency work mainly compresses tool observations or routes selected turns to cheaper models, but the practical opportunity for these optimizations remains unclear. We analyze 600 publicly released coding-agent trajectories spanning two agent scaffolds and two language models. Median observations contain only 118-286 tokens, while the large build logs commonly used to motivate compression occur near the ninety-ninth percentile. Moreover, 98.6% of input tokens are served from a prefix cache, so flat input pricing overestimates loop cost by roughly 7x on average under near-ideal caching.
We derive and empirically validate a closed-form break-even condition for model delegation. Delegation increases cost in half of the evaluated settings, near the predicted boundary. More broadly, turn count explains 84% of the variance in log trajectory cost, compared with 4% for mean observation size; the most expensive trajectories have the most turns despite having the smallest observations. These associations suggest that observation-focused methods target a secondary cost factor, and identify unnecessary turns as the larger and largely unexamined opportunity, though we do not establish that turns can be removed while holding task success fixed. We release TurnCost, a tool for re-scoring trajectory corpora under alternative caching, pricing, compression, and routing policies.
Article Search
Artifacts Available
Article: asews26povcmain-p15-p (type: Full Paper) doi:10.1145/3843779.3844635
POVC '26: "Toward Trace-Guided Automated ..."
Toward Trace-Guided Automated Program Repair
David Georg Reichelt, Shinhyung Yang, and Wilhelm Hasselbring
(Lancaster University Leipzig, Germany; Universität Leipzig, Germany; Kiel University, Germany)
Fixing bugs is a time-consuming task. Researchers and developers hence aim to automate program repair (APR) as much as possible. Recent advances in Large Language Models (LLMs) made automated fixing of bugs widely accessible to practitioners. However, while LLMs are in principle capable of generating code and fixing programs, their ability to locate and therefore effectively fix bugs is limited by their context length. To guide them towards the parts of the program that require fixing, LLMs could be prompted with information from dynamic analysis techniques.
We expect that the most efficient way to help LLMs debug code is by providing the LLM with information to interactively find the bug location. To do so, we propose two steps: Tailoring the prompt with trace information and giving the LLM a tool call to get further trace information. This trace information needs to be preprocessed in a way that it can be efficiently used by the LLM and makes it possible to understand the program behavior.
In this paper, we describe our concept of an APR approach that is guided by tracing data and which is therefore capable of fixing bugs in huge codebases. As a proof of concept, we show that in the bug dataset defects4j, location information achieves an absolute increase of up to 22.4 % in fixed bugs with Gemma-4-31B compared to a baseline without location information. Based on our concept and our proof of concept, we outline the research agenda that is necessary to enable trace-guided APR.
Article Search
Article: asews26povcmain-p4-p (type: Full Paper) doi:10.1145/3843779.3844631
POVC '26: "Agent4RE: A Self-Refining ..."
Agent4RE: A Self-Refining Multi-agent Framework for End-to-End Software Requirements Engineering and Benchmarking
Yongjian Tang, Linhan Li, and Thomas Runkler
(Siemens, Germany; TU Munich, Germany)
Existing LLM-based approaches for software Requirements Engineering (RE) typically rely on basic prompting strategies or rudimentary agent collaboration, under-utilizing the full potential of multi-agent systems. Meanwhile, available datasets focus on isolated subtasks, such as requirements extraction, classification, and completeness detection, leaving the absence of an end-to-end RE benchmark that spans from requirements elicitation to generation. We present Agent4RE - a self-refining multi-agent RE system that orchestrates specialized agents and incorporates two iterative improvement loops. To support evaluation, we construct RE-E2E - a real-world dataset built from human-written requirement specifications, enabling end-to-end assessment of RE workflows. Building on this foundation, we further propose two enhanced Agent4RE versions that incorporate either autonomous self-refinement or structured human feedback, and analyze their strengths and limitations across different scenarios. Evaluation on 8 Large Language Models (LLMs) demonstrates that all three Agent4RE variants consistently outperform a domain-context-augmented prompting baseline by average 8% in text-based metrics. The two enhanced variants achieve the highest LLM-as-a-judge and human ratings, surpassing two RE baselines by approximately 0.8 points on a four-point scale. This consistent performance establishes Agent4RE as a practical end-to-end RE solution for industrial environments.
Article Search
Article: asews26povcmain-p11-p (type: Full Paper) doi:10.1145/3843779.3844634
POVC '26: "Prompt Pipelines as Engineered ..."
Prompt Pipelines as Engineered Artifacts: Regression-Testing a Multi-agent Threat-Modeling Pipeline
Eldar Mametov, Andrey Sadovykh, Eugene Zouev, and Alberto Sillitti
(Innopolis University, Russia; SOFTEAM, France; Centre for Applied Software Engineering, Italy)
Security pipelines built on large language models (LLMs) are, at their core, collections of prompts: role instructions, retrieval queries, critique templates, and output formats. In practice these prompts evolve the way early software did: by trial and error, with changes that are rarely documented and never regression-tested. This paper reports engineering experience from treating a multi-agent threat-modeling pipeline as a set of versioned, tested software artifacts. The pipeline decomposes STRIDE analysis into six specialised agent prompts with retrieval enrichment and a critique stage; a fixed corpus of 24 system architectures with expert-curated reference threats serves as a regression oracle for every change to a prompt or pipeline stage. We describe three documented design decisions the harness drove: removing a chunk-filtered retrieval stage that consumed tokens without improving coverage, declining to adopt the most expensive pipeline as the default after the oracle showed a six-agent decomposition matched its coverage at roughly one-ninth the token cost, and catching a metric-specification drift between documentation and implementation. We distil the recurring failure modes (retrieval-induced specificity bias, mechanism-versus-impact category confusion, and systematic risk over-estimation) into checks that now guard the pipeline. The experience suggests that the PromptOps practices recently proposed for general LLM applications transfer directly to security engineering, and that a fixed evaluation corpus with cost-aware metrics is the single most valuable artifact a prompt-pipeline team can maintain.
Article Search
Artifacts Available
Article: asews26povcmain-p10-p (type: Full Paper) doi:10.1145/3843779.3844633
POVC '26: "Geometry-Aware Prompt Context ..."
Geometry-Aware Prompt Context for Model Consistency Management
Monalisha Ojha, Shilpi Gupta, and Rahul Sharma
(University of Mannheim, Germany; Tallinn University of Technology, Estonia)
Keeping independently developed models consistent with a large language model means prompting it with two metamodels at once, yet industrial metamodels hold hundreds of classifiers and cannot be pasted in. The usual remedy---serialise, chunk, and retrieve by text or embedding similarity---flattens inheritance, containment and typed references into linear text, so retrieval compares names rather than structure, and fails when two teams name the same concept differently. A metamodel already carries an exact metric: its hierarchy induces a geodesic distance that needs no training and that an embedding can only approximate. That metric holds within one metamodel, not across two; what joins them is a few correspondences confirmed by an engineer. We present Anchored Geometric Propagation (AGP): it joins two metamodels at these anchors, ranks candidates by geodesic distance blended with name similarity at a weight calibrated on the anchors, promotes confident matches to new anchors, and defers the rest to the engineer. Selected classifiers carry their supertypes and feature types, so the prompt holds a well-formed fragment of the hierarchy. On metamodel pairs derived from ModelSet, AGP covers the large majority of queries at a fraction of the cost of full serialisation, and outperforms lexical retrieval precisely where lexical retrieval fails. The evaluation is intrinsic: it measures what reaches the prompt, not what a language model subsequently does with it, and so establishes a necessary condition for the downstream task rather than an accuracy gain.
Article Search
Article: asews26povcmain-p20-p (type: Full Paper) doi:10.1145/3843779.3844638
POVC '26: "Prompt Pipelines as Engineered ..."
Prompt Pipelines as Engineered Artifacts: Regression-Testing a Multi-agent Threat-Modeling Pipeline
Eldar Mametov, Andrey Sadovykh, Eugene Zouev, and Alberto Sillitti
(Innopolis University, Russia; SOFTEAM, France; Centre for Applied Software Engineering, Italy)
Security pipelines built on large language models (LLMs) are, at their core, collections of prompts: role instructions, retrieval queries, critique templates, and output formats. In practice these prompts evolve the way early software did: by trial and error, with changes that are rarely documented and never regression-tested. This paper reports engineering experience from treating a multi-agent threat-modeling pipeline as a set of versioned, tested software artifacts. The pipeline decomposes STRIDE analysis into six specialised agent prompts with retrieval enrichment and a critique stage; a fixed corpus of 24 system architectures with expert-curated reference threats serves as a regression oracle for every change to a prompt or pipeline stage. We describe three documented design decisions the harness drove: removing a chunk-filtered retrieval stage that consumed tokens without improving coverage, declining to adopt the most expensive pipeline as the default after the oracle showed a six-agent decomposition matched its coverage at roughly one-ninth the token cost, and catching a metric-specification drift between documentation and implementation. We distil the recurring failure modes (retrieval-induced specificity bias, mechanism-versus-impact category confusion, and systematic risk over-estimation) into checks that now guard the pipeline. The experience suggests that the PromptOps practices recently proposed for general LLM applications transfer directly to security engineering, and that a fixed evaluation corpus with cost-aware metrics is the single most valuable artifact a prompt-pipeline team can maintain.
Article Search
Artifacts Available
Article: asews26povcmain-p10-p (type: Full Paper) doi:10.1145/3843779.3844633
POVC '26: "Prompts Matter: On the Impact ..."
Prompts Matter: On the Impact of Code Smell Remediation with Large Language Models
Nadim Hagatulah, Markus Borg, and Emma Söderberg
(Lund University, Sweden; CodeScene, Sweden)
Code smells accumulate as technical debt, reducing software maintainability. Refactoring is the established remediation practice, but it is complex and often deprioritized. Large Language Models can support automated refactoring, but their effectiveness depends on how they are prompted. In this study, we evaluate 14 prompt configurations with GPT-OSS and Qwen3 on 2,000 Java and Python functions, producing 168,000 function-level refactorings. We evaluate behavior preservation, code smell remediation, and cost by token proxy. Results show that prompt effectiveness is contextual. For GPT-OSS, prompts combining explicit smell thresholds with reasoning triggers increased remediation odds by up to 1.32× over zero-shot at a 1.05–1.10× cost multiplier. For Qwen3, prompts eliciting reasoning reduced behavior-preservation odds to 0.75× of zero-shot and raised cost by up to 1.51× zero-shot. Conversely, the threshold-only prompt increased remediation odds by up to 1.18× with near-neutral overhead (1.01–1.02×). Cost per successful refactoring was 1.01–1.04× ZS for GPT-OSS and 1.12–1.21× for Qwen3. Overall, prompt choice matters, but model choice produced the larger difference. These findings support validation-centered, model-specific prompt selection rather than a uniform strategy.
Article Search
Article: asews26povcmain-p18-p (type: Full Paper) doi:10.1145/3843779.3844637
POVC '26: "VibeCheck: Assessing the Quality ..."
VibeCheck: Assessing the Quality of LLM-Generated Unit Tests: A Multi-agent Empirical Study across Heterogeneous Repositories
Anika Tabassum, Mushahid Intesum, Md. Fahim Arefin, and Tarannum Shaila Zaman
(University of Dhaka, Bangladesh; University of Maryland, Baltimore County, USA)
LLM-based IDE agents are increasingly used to generate repository-grounded unit tests, yet common evaluations often rely on execution success or coverage. These metrics can miss deeper quality issues such as weak assertions, missing edge cases, poor isolation, and limited maintainability. This paper presents VibeCheck, an empirical study of unit test generation across 15 student-developed Python and JavaScript/TypeScript repositories. We evaluate Kiro, Antigravity, and Cursor with Claude Sonnet 4.5 as the underlying agent, under repository-only, zero-shot conditions using a five-dimensional rubric covering runnability, assertion strength, logic and edge-case coverage, isolation/determinism, and maintainability. We also apply leave-one-out cross-agent peer evaluation to compare tools and identify failure patterns. Results show a clear execution-adequacy gap: generated tests are often runnable but frequently lack strong assertions and meaningful behavioral coverage. Weak assertions and missing edge cases occur more often than blocking failures, showing that runnable tests can still be shallow. VibeCheck provides a reliability-oriented framework for evaluating LLM-generated tests beyond pass/fail outcomes.
Article Search
Article: asews26povcmain-p7-p (type: Full Paper) doi:10.1145/3843779.3844632
POVC '26: "Agent4RE: A Self-Refining ..."
Agent4RE: A Self-Refining Multi-agent Framework for End-to-End Software Requirements Engineering and Benchmarking
Yongjian Tang, Linhan Li, and Thomas Runkler
(Siemens, Germany; TU Munich, Germany)
Existing LLM-based approaches for software Requirements Engineering (RE) typically rely on basic prompting strategies or rudimentary agent collaboration, under-utilizing the full potential of multi-agent systems. Meanwhile, available datasets focus on isolated subtasks, such as requirements extraction, classification, and completeness detection, leaving the absence of an end-to-end RE benchmark that spans from requirements elicitation to generation. We present Agent4RE - a self-refining multi-agent RE system that orchestrates specialized agents and incorporates two iterative improvement loops. To support evaluation, we construct RE-E2E - a real-world dataset built from human-written requirement specifications, enabling end-to-end assessment of RE workflows. Building on this foundation, we further propose two enhanced Agent4RE versions that incorporate either autonomous self-refinement or structured human feedback, and analyze their strengths and limitations across different scenarios. Evaluation on 8 Large Language Models (LLMs) demonstrates that all three Agent4RE variants consistently outperform a domain-context-augmented prompting baseline by average 8% in text-based metrics. The two enhanced variants achieve the highest LLM-as-a-judge and human ratings, surpassing two RE baselines by approximately 0.8 points on a four-point scale. This consistent performance establishes Agent4RE as a practical end-to-end RE solution for industrial environments.
Article Search
Article: asews26povcmain-p11-p (type: Full Paper) doi:10.1145/3843779.3844634
POVC '26: "Toward Trace-Guided Automated ..."
Toward Trace-Guided Automated Program Repair
David Georg Reichelt, Shinhyung Yang, and Wilhelm Hasselbring
(Lancaster University Leipzig, Germany; Universität Leipzig, Germany; Kiel University, Germany)
Fixing bugs is a time-consuming task. Researchers and developers hence aim to automate program repair (APR) as much as possible. Recent advances in Large Language Models (LLMs) made automated fixing of bugs widely accessible to practitioners. However, while LLMs are in principle capable of generating code and fixing programs, their ability to locate and therefore effectively fix bugs is limited by their context length. To guide them towards the parts of the program that require fixing, LLMs could be prompted with information from dynamic analysis techniques.
We expect that the most efficient way to help LLMs debug code is by providing the LLM with information to interactively find the bug location. To do so, we propose two steps: Tailoring the prompt with trace information and giving the LLM a tool call to get further trace information. This trace information needs to be preprocessed in a way that it can be efficiently used by the LLM and makes it possible to understand the program behavior.
In this paper, we describe our concept of an APR approach that is guided by tracing data and which is therefore capable of fixing bugs in huge codebases. As a proof of concept, we show that in the bug dataset defects4j, location information achieves an absolute increase of up to 22.4 % in fixed bugs with Gemma-4-31B compared to a baseline without location information. Based on our concept and our proof of concept, we outline the research agenda that is necessary to enable trace-guided APR.
Article Search
Article: asews26povcmain-p4-p (type: Full Paper) doi:10.1145/3843779.3844631
POVC '26: "VibeCheck: Assessing the Quality ..."
VibeCheck: Assessing the Quality of LLM-Generated Unit Tests: A Multi-agent Empirical Study across Heterogeneous Repositories
Anika Tabassum, Mushahid Intesum, Md. Fahim Arefin, and Tarannum Shaila Zaman
(University of Dhaka, Bangladesh; University of Maryland, Baltimore County, USA)
LLM-based IDE agents are increasingly used to generate repository-grounded unit tests, yet common evaluations often rely on execution success or coverage. These metrics can miss deeper quality issues such as weak assertions, missing edge cases, poor isolation, and limited maintainability. This paper presents VibeCheck, an empirical study of unit test generation across 15 student-developed Python and JavaScript/TypeScript repositories. We evaluate Kiro, Antigravity, and Cursor with Claude Sonnet 4.5 as the underlying agent, under repository-only, zero-shot conditions using a five-dimensional rubric covering runnability, assertion strength, logic and edge-case coverage, isolation/determinism, and maintainability. We also apply leave-one-out cross-agent peer evaluation to compare tools and identify failure patterns. Results show a clear execution-adequacy gap: generated tests are often runnable but frequently lack strong assertions and meaningful behavioral coverage. Weak assertions and missing edge cases occur more often than blocking failures, showing that runnable tests can still be shallow. VibeCheck provides a reliability-oriented framework for evaluating LLM-generated tests beyond pass/fail outcomes.
Article Search
Article: asews26povcmain-p7-p (type: Full Paper) doi:10.1145/3843779.3844632
POVC '26: "Prompt Pipelines as Engineered ..."
Prompt Pipelines as Engineered Artifacts: Regression-Testing a Multi-agent Threat-Modeling Pipeline
Eldar Mametov, Andrey Sadovykh, Eugene Zouev, and Alberto Sillitti
(Innopolis University, Russia; SOFTEAM, France; Centre for Applied Software Engineering, Italy)
Security pipelines built on large language models (LLMs) are, at their core, collections of prompts: role instructions, retrieval queries, critique templates, and output formats. In practice these prompts evolve the way early software did: by trial and error, with changes that are rarely documented and never regression-tested. This paper reports engineering experience from treating a multi-agent threat-modeling pipeline as a set of versioned, tested software artifacts. The pipeline decomposes STRIDE analysis into six specialised agent prompts with retrieval enrichment and a critique stage; a fixed corpus of 24 system architectures with expert-curated reference threats serves as a regression oracle for every change to a prompt or pipeline stage. We describe three documented design decisions the harness drove: removing a chunk-filtered retrieval stage that consumed tokens without improving coverage, declining to adopt the most expensive pipeline as the default after the oracle showed a six-agent decomposition matched its coverage at roughly one-ninth the token cost, and catching a metric-specification drift between documentation and implementation. We distil the recurring failure modes (retrieval-induced specificity bias, mechanism-versus-impact category confusion, and systematic risk over-estimation) into checks that now guard the pipeline. The experience suggests that the PromptOps practices recently proposed for general LLM applications transfer directly to security engineering, and that a fixed evaluation corpus with cost-aware metrics is the single most valuable artifact a prompt-pipeline team can maintain.
Article Search
Artifacts Available
Article: asews26povcmain-p10-p (type: Full Paper) doi:10.1145/3843779.3844633