LMPL '26: "Authority Is Not a String: ..."
Authority Is Not a String: A Capability-Scoped Harness for Prompt-Injection-Resistant Coding Agents
Dimitrios Stamatios Bouras, Yihan Dai, and Sergey Mechtaev
(Peking University, China)
Coding agents use system-level tools to read files, execute commands, and modify source code. Within the agent’s sandbox, these tools often carry ambient authority: naming a resource is sufficient to act on it. Indirect prompt injection exploits this authority by placing instructions in repository files or tool output that cause the agent to perform actions the user did not request.
We propose CapScope, a harness-level authorization mechanism that restricts tool use without requiring the model to identify malicious text. Before repository contents or tool output are read, CapScope derives a task-wide authority ceiling from trusted input. It then assigns each agent a separate set of typed capabilities, stored outside the model’s context. Every tool call is checked against the capabilities of the agent that issued it. Permissions assigned to one sub-agent are therefore not automatically available to another. An injection may cause an agent to request an action, but the request is blocked unless that agent already has the required capability.
We implement CapScope on the Pi coding agent and evaluate it in a repair workflow where an orchestrator delegates subtasks to separate sub-agents. The evaluation covers five Python tasks, five injection surfaces, four authorization conditions, and three trials per cell (300 runs). The injected effect executes in 33–47/75 runs under the ambient-authority and global-policy baselines, compared with 3/75 under CapScope. CapScope completes 68/75 repairs, while the baselines complete 68–72/75.
Article Search
Article: splashws26lmplmain-p72-p (type: Full Paper) doi:10.1145/3843750.3843843
LMPL '26: "Invariant Pretraining for ..."
Invariant Pretraining for Robust Code Representations
Yifeng He, Yundi Xu, Christopher Castro Gaw Gonzalo, Zili Wang, and Hao Chen
(University of California at Davis, USA; University of Hong Kong, China)
Encoder-based code representation models remain widely used for discriminative tasks such as clone detection and code classification because their small size and low inference cost matter. Yet on invariant programs, semantically equivalent code with different syntax, their representations degrade substantially despite unchanged behavior. We measure this gap across four encoder baselines, two tasks, and four datasets, then test a minimal code-only continued pretraining recipe that consistently recovers part of it. Invariant pretraining (InvPT) applies semantics-preserving transformations and combines masked language modeling with multi-positive supervised contrastive learning. All augmentations of a source function are positives: self-contrast pairs use the same code with different masks, whereas invariant-contrast pairs use transformed code, providing varied difficulty without paired natural-language data. Across all model-dataset comparisons, InvPT improves robustness on transformed test sets by a median of 8.1 percentage points for clone detection (up to 11.0) and 3.6 points for code classification (up to 19.2), while matching or improving standard performance. Ablations identify multi-positive invariant contrast as the main source of these gains. Because the test transformations are composed from the pretraining operator family, the results establish invariance only to that family, not robustness in general. Thus, the contribution is not a new objective but a measurement of where encoder robustness breaks and how much of it a simple code-only recipe recovers.
Article Search
Article: splashws26lmplmain-p19-p (type: Full Paper) doi:10.1145/3843750.3843840
LMPL '26: "FlowCheck: Helping End-Users ..."
FlowCheck: Helping End-Users Specify and Verify Intent in Vibe-Coded Web Apps
Reya Vir, Lydia Chilton, Zhuo Zhang, and Eugene Wu
(Columbia University, USA)
Vibe-coded applications often contain silent behavioral failures in which the interface appears functional even though user-visible information does not flow to the expected state or output. We introduce FlowCheck, a constraint language to specify these user-visible information flows directly through the application interface, where constraints can also be displayed and inspected without reading code, and are structured enough for reliable LLM generation. FlowCheck translates the constraints into deterministic CodeQL analyses, and we evaluate it across four applications generated via Claude Code, and compare with three coding models as bug-finding baselines. We find that FlowCheck correctly translates and flags all 30 of our injected constraint violations with no false positives. In contrast, frontier models (Claude Opus 4.7, DeepSeek V3, and Gemini Pro) showed significantly lower accuracy when prompted to find bugs in the same code, with none achieving full accuracy. This approach lets vibe coders state intent in terms of the interface they understand, and checks it deterministically against the code they do not.
Article Search
Article: splashws26lmplmain-p81-p (type: Full Paper) doi:10.1145/3843750.3843844
LMPL '26: "Authority Is Not a String: ..."
Authority Is Not a String: A Capability-Scoped Harness for Prompt-Injection-Resistant Coding Agents
Dimitrios Stamatios Bouras, Yihan Dai, and Sergey Mechtaev
(Peking University, China)
Coding agents use system-level tools to read files, execute commands, and modify source code. Within the agent’s sandbox, these tools often carry ambient authority: naming a resource is sufficient to act on it. Indirect prompt injection exploits this authority by placing instructions in repository files or tool output that cause the agent to perform actions the user did not request.
We propose CapScope, a harness-level authorization mechanism that restricts tool use without requiring the model to identify malicious text. Before repository contents or tool output are read, CapScope derives a task-wide authority ceiling from trusted input. It then assigns each agent a separate set of typed capabilities, stored outside the model’s context. Every tool call is checked against the capabilities of the agent that issued it. Permissions assigned to one sub-agent are therefore not automatically available to another. An injection may cause an agent to request an action, but the request is blocked unless that agent already has the required capability.
We implement CapScope on the Pi coding agent and evaluate it in a repair workflow where an orchestrator delegates subtasks to separate sub-agents. The evaluation covers five Python tasks, five injection surfaces, four authorization conditions, and three trials per cell (300 runs). The injected effect executes in 33–47/75 runs under the ambient-authority and global-policy baselines, compared with 3/75 under CapScope. CapScope completes 68/75 repairs, while the baselines complete 68–72/75.
Article Search
Article: splashws26lmplmain-p72-p (type: Full Paper) doi:10.1145/3843750.3843843
LMPL '26: "Detecting Database Migration ..."
Detecting Database Migration Non-equivalence through LLM-Assisted Mutation-Score-Guided Testing
Yuhao Zhang, Sayan Ghosh, Amit Kachroo, and Jun Huan
(AWS AI Labs, USA)
Database migration involves translating schemas, data, and queries from one database system to another. Query translation, a key step in this process, is widely practiced but error-prone: subtle semantic differences between SQL dialects can cause silent data corruption and incorrect query results. As migrations increasingly rely on LLMs, the translated queries lack equivalence validations. Existing formal verification tools cannot handle the full range of features in SQL dialects, and differential testing provides no confidence measure when all tests pass. We propose LAMT, an LLM-assisted mutation-score-guided testing approach that addresses both problems. Given a pair of pre-migration and post-migration SQL queries, LAMT operates as an adversarial loop: an LLM table generator produces test data aimed at killing query mutants, i.e., exposing behavioral differences, while an LLM query generator produces mutants that attempt to evade detection by the current test suite. The resulting mutation score is then calibrated into a confidence threshold that estimates the risk of undetected non-equivalent pairs. We evaluate LAMT on 1,035 cross-dialect migration pairs spanning three dialect pairs. Query-aware table generation reduces the average false negative rate from 58.3% to 20.0% with 4 more generated tables. The mutation score calibration provides directional confidence estimates, though a gap between estimated and true FNR remains, particularly for weaker models.
Article Search
Article: splashws26lmplmain-p85-p (type: Full Paper) doi:10.1145/3843750.3843845
LMPL '26: "Predicting Program Exit Code ..."
Predicting Program Exit Code with LLMs and Programming Language Semantics
Lara Marinov, Aditya Thimmaiah, Jayanth Srinivasa, Junyi Jessy Li, and Milos Gligoric
(University of Texas at Austin, USA; Cisco, USA)
Large language models (LLMs) have shown proficiency in various software engineering tasks, such as code generation and translation. However, a key limitation in their performance may be their (lack of) understanding of programming-language semantics. Even when explicit semantics are given, it remains unclear whether LLMs apply those rules or lean on priors learned during pre-training instead. We study if LLMs lean on priors or given semantics with a novel task–Program Executability Prediction (PrEx)–that asks models to predict whether a program is semantically valid or invalid (and, if invalid, which formal rule it violates) given the program’s syntax and operational semantics. Because PrEx requires both valid and invalid programs, we build a dataset with systematically generated invalid transformations derived from valid programs. We evaluate open-source coding LLMs under two semantic formalisms and two semantic shifts across Human-Written, LLM-Translated, and Fuzzer-Generated program splits. Our findings show that LLMs lean on pre-training priors rather than systematically applying the given rules, performing especially poorly on modified semantics and degrading further as program complexity increases. PrEx is available at https://github.com/EngineeringSoftware/prex.
Article Search
Article: splashws26lmplmain-p42-p (type: Full Paper) doi:10.1145/3843750.3843842
LMPL '26: "Invariant Pretraining for ..."
Invariant Pretraining for Robust Code Representations
Yifeng He, Yundi Xu, Christopher Castro Gaw Gonzalo, Zili Wang, and Hao Chen
(University of California at Davis, USA; University of Hong Kong, China)
Encoder-based code representation models remain widely used for discriminative tasks such as clone detection and code classification because their small size and low inference cost matter. Yet on invariant programs, semantically equivalent code with different syntax, their representations degrade substantially despite unchanged behavior. We measure this gap across four encoder baselines, two tasks, and four datasets, then test a minimal code-only continued pretraining recipe that consistently recovers part of it. Invariant pretraining (InvPT) applies semantics-preserving transformations and combines masked language modeling with multi-positive supervised contrastive learning. All augmentations of a source function are positives: self-contrast pairs use the same code with different masks, whereas invariant-contrast pairs use transformed code, providing varied difficulty without paired natural-language data. Across all model-dataset comparisons, InvPT improves robustness on transformed test sets by a median of 8.1 percentage points for clone detection (up to 11.0) and 3.6 points for code classification (up to 19.2), while matching or improving standard performance. Ablations identify multi-positive invariant contrast as the main source of these gains. Because the test transformations are composed from the pretraining operator family, the results establish invariance only to that family, not robustness in general. Thus, the contribution is not a new objective but a measurement of where encoder robustness breaks and how much of it a simple code-only recipe recovers.
Article Search
Article: splashws26lmplmain-p19-p (type: Full Paper) doi:10.1145/3843750.3843840
LMPL '26: "Invariant Pretraining for ..."
Invariant Pretraining for Robust Code Representations
Yifeng He, Yundi Xu, Christopher Castro Gaw Gonzalo, Zili Wang, and Hao Chen
(University of California at Davis, USA; University of Hong Kong, China)
Encoder-based code representation models remain widely used for discriminative tasks such as clone detection and code classification because their small size and low inference cost matter. Yet on invariant programs, semantically equivalent code with different syntax, their representations degrade substantially despite unchanged behavior. We measure this gap across four encoder baselines, two tasks, and four datasets, then test a minimal code-only continued pretraining recipe that consistently recovers part of it. Invariant pretraining (InvPT) applies semantics-preserving transformations and combines masked language modeling with multi-positive supervised contrastive learning. All augmentations of a source function are positives: self-contrast pairs use the same code with different masks, whereas invariant-contrast pairs use transformed code, providing varied difficulty without paired natural-language data. Across all model-dataset comparisons, InvPT improves robustness on transformed test sets by a median of 8.1 percentage points for clone detection (up to 11.0) and 3.6 points for code classification (up to 19.2), while matching or improving standard performance. Ablations identify multi-positive invariant contrast as the main source of these gains. Because the test transformations are composed from the pretraining operator family, the results establish invariance only to that family, not robustness in general. Thus, the contribution is not a new objective but a measurement of where encoder robustness breaks and how much of it a simple code-only recipe recovers.
Article Search
Article: splashws26lmplmain-p19-p (type: Full Paper) doi:10.1145/3843750.3843840
LMPL '26: "Detecting Database Migration ..."
Detecting Database Migration Non-equivalence through LLM-Assisted Mutation-Score-Guided Testing
Yuhao Zhang, Sayan Ghosh, Amit Kachroo, and Jun Huan
(AWS AI Labs, USA)
Database migration involves translating schemas, data, and queries from one database system to another. Query translation, a key step in this process, is widely practiced but error-prone: subtle semantic differences between SQL dialects can cause silent data corruption and incorrect query results. As migrations increasingly rely on LLMs, the translated queries lack equivalence validations. Existing formal verification tools cannot handle the full range of features in SQL dialects, and differential testing provides no confidence measure when all tests pass. We propose LAMT, an LLM-assisted mutation-score-guided testing approach that addresses both problems. Given a pair of pre-migration and post-migration SQL queries, LAMT operates as an adversarial loop: an LLM table generator produces test data aimed at killing query mutants, i.e., exposing behavioral differences, while an LLM query generator produces mutants that attempt to evade detection by the current test suite. The resulting mutation score is then calibrated into a confidence threshold that estimates the risk of undetected non-equivalent pairs. We evaluate LAMT on 1,035 cross-dialect migration pairs spanning three dialect pairs. Query-aware table generation reduces the average false negative rate from 58.3% to 20.0% with 4 more generated tables. The mutation score calibration provides directional confidence estimates, though a gap between estimated and true FNR remains, particularly for weaker models.
Article Search
Article: splashws26lmplmain-p85-p (type: Full Paper) doi:10.1145/3843750.3843845
LMPL '26: "Detecting Database Migration ..."
Detecting Database Migration Non-equivalence through LLM-Assisted Mutation-Score-Guided Testing
Yuhao Zhang, Sayan Ghosh, Amit Kachroo, and Jun Huan
(AWS AI Labs, USA)
Database migration involves translating schemas, data, and queries from one database system to another. Query translation, a key step in this process, is widely practiced but error-prone: subtle semantic differences between SQL dialects can cause silent data corruption and incorrect query results. As migrations increasingly rely on LLMs, the translated queries lack equivalence validations. Existing formal verification tools cannot handle the full range of features in SQL dialects, and differential testing provides no confidence measure when all tests pass. We propose LAMT, an LLM-assisted mutation-score-guided testing approach that addresses both problems. Given a pair of pre-migration and post-migration SQL queries, LAMT operates as an adversarial loop: an LLM table generator produces test data aimed at killing query mutants, i.e., exposing behavioral differences, while an LLM query generator produces mutants that attempt to evade detection by the current test suite. The resulting mutation score is then calibrated into a confidence threshold that estimates the risk of undetected non-equivalent pairs. We evaluate LAMT on 1,035 cross-dialect migration pairs spanning three dialect pairs. Query-aware table generation reduces the average false negative rate from 58.3% to 20.0% with 4 more generated tables. The mutation score calibration provides directional confidence estimates, though a gap between estimated and true FNR remains, particularly for weaker models.
Article Search
Article: splashws26lmplmain-p85-p (type: Full Paper) doi:10.1145/3843750.3843845
LMPL '26: "Semantic Prefix Oracles for ..."
Semantic Prefix Oracles for LLM Decoding: Contracts and Differential Validation
Paul Kronlund-Drouault
(Unsuspicious Industries, France)
Constrained decoding can enforce regular or context-free output formats, but many program-generation failures are semantic: scope, typing, and declaration effects depend on context. We present semantic grammar specifications, a declarative formalism that attaches such constraints to a context-free surface and executes them during Earley descent. Our implementation enforces safe pruning: it rejects only prefixes whose semantic contradictions cannot be repaired by any continuation. A separate, grammar-dependent, dead-end freedom property guarantees the existence of a realizable witness for each remaining branch. We give simple sufficient conditions based on surface productivity, type coverage, and left-to-right constraint flow. Our finite-lambda, core ML, and C-like fragments satisfy them, while the STLC instance used in our experiments does not: plain STLC can violate type coverage, and we show how restricting its type universe recovers it. A tokenizer-lifting lemma carries character-level witnesses to token sequences under an explicit vocabulary-coverage hypothesis.
We validate the implementation differentially against production compilers (ocamlc, cc). Across every prefix of 65 compiler-valid programs we observe zero false prunes. The semantic oracle localizes 25/30 invalid programs mid-stream, against 0/30 for a syntax-only oracle, and agrees on 42/42 recursion probes. A twelve-model generation study, including a matched semantic-versus-syntactic ablation for nine models, finds nonnegative observed semantic-minus-syntactic point estimates for every model-language pair, with maxima of +15.2 points on STLC task correctness and +14.3 points on ML validity.
Article Search
Article: splashws26lmplmain-p30-p (type: Full Paper) doi:10.1145/3843750.3843841
LMPL '26: "Predicting Program Exit Code ..."
Predicting Program Exit Code with LLMs and Programming Language Semantics
Lara Marinov, Aditya Thimmaiah, Jayanth Srinivasa, Junyi Jessy Li, and Milos Gligoric
(University of Texas at Austin, USA; Cisco, USA)
Large language models (LLMs) have shown proficiency in various software engineering tasks, such as code generation and translation. However, a key limitation in their performance may be their (lack of) understanding of programming-language semantics. Even when explicit semantics are given, it remains unclear whether LLMs apply those rules or lean on priors learned during pre-training instead. We study if LLMs lean on priors or given semantics with a novel task–Program Executability Prediction (PrEx)–that asks models to predict whether a program is semantically valid or invalid (and, if invalid, which formal rule it violates) given the program’s syntax and operational semantics. Because PrEx requires both valid and invalid programs, we build a dataset with systematically generated invalid transformations derived from valid programs. We evaluate open-source coding LLMs under two semantic formalisms and two semantic shifts across Human-Written, LLM-Translated, and Fuzzer-Generated program splits. Our findings show that LLMs lean on pre-training priors rather than systematically applying the given rules, performing especially poorly on modified semantics and degrading further as program complexity increases. PrEx is available at https://github.com/EngineeringSoftware/prex.
Article Search
Article: splashws26lmplmain-p42-p (type: Full Paper) doi:10.1145/3843750.3843842
LMPL '26: "Predicting Program Exit Code ..."
Predicting Program Exit Code with LLMs and Programming Language Semantics
Lara Marinov, Aditya Thimmaiah, Jayanth Srinivasa, Junyi Jessy Li, and Milos Gligoric
(University of Texas at Austin, USA; Cisco, USA)
Large language models (LLMs) have shown proficiency in various software engineering tasks, such as code generation and translation. However, a key limitation in their performance may be their (lack of) understanding of programming-language semantics. Even when explicit semantics are given, it remains unclear whether LLMs apply those rules or lean on priors learned during pre-training instead. We study if LLMs lean on priors or given semantics with a novel task–Program Executability Prediction (PrEx)–that asks models to predict whether a program is semantically valid or invalid (and, if invalid, which formal rule it violates) given the program’s syntax and operational semantics. Because PrEx requires both valid and invalid programs, we build a dataset with systematically generated invalid transformations derived from valid programs. We evaluate open-source coding LLMs under two semantic formalisms and two semantic shifts across Human-Written, LLM-Translated, and Fuzzer-Generated program splits. Our findings show that LLMs lean on pre-training priors rather than systematically applying the given rules, performing especially poorly on modified semantics and degrading further as program complexity increases. PrEx is available at https://github.com/EngineeringSoftware/prex.
Article Search
Article: splashws26lmplmain-p42-p (type: Full Paper) doi:10.1145/3843750.3843842
LMPL '26: "Authority Is Not a String: ..."
Authority Is Not a String: A Capability-Scoped Harness for Prompt-Injection-Resistant Coding Agents
Dimitrios Stamatios Bouras, Yihan Dai, and Sergey Mechtaev
(Peking University, China)
Coding agents use system-level tools to read files, execute commands, and modify source code. Within the agent’s sandbox, these tools often carry ambient authority: naming a resource is sufficient to act on it. Indirect prompt injection exploits this authority by placing instructions in repository files or tool output that cause the agent to perform actions the user did not request.
We propose CapScope, a harness-level authorization mechanism that restricts tool use without requiring the model to identify malicious text. Before repository contents or tool output are read, CapScope derives a task-wide authority ceiling from trusted input. It then assigns each agent a separate set of typed capabilities, stored outside the model’s context. Every tool call is checked against the capabilities of the agent that issued it. Permissions assigned to one sub-agent are therefore not automatically available to another. An injection may cause an agent to request an action, but the request is blocked unless that agent already has the required capability.
We implement CapScope on the Pi coding agent and evaluate it in a repair workflow where an orchestrator delegates subtasks to separate sub-agents. The evaluation covers five Python tasks, five injection surfaces, four authorization conditions, and three trials per cell (300 runs). The injected effect executes in 33–47/75 runs under the ambient-authority and global-policy baselines, compared with 3/75 under CapScope. CapScope completes 68/75 repairs, while the baselines complete 68–72/75.
Article Search
Article: splashws26lmplmain-p72-p (type: Full Paper) doi:10.1145/3843750.3843843
LMPL '26: "Predicting Program Exit Code ..."
Predicting Program Exit Code with LLMs and Programming Language Semantics
Lara Marinov, Aditya Thimmaiah, Jayanth Srinivasa, Junyi Jessy Li, and Milos Gligoric
(University of Texas at Austin, USA; Cisco, USA)
Large language models (LLMs) have shown proficiency in various software engineering tasks, such as code generation and translation. However, a key limitation in their performance may be their (lack of) understanding of programming-language semantics. Even when explicit semantics are given, it remains unclear whether LLMs apply those rules or lean on priors learned during pre-training instead. We study if LLMs lean on priors or given semantics with a novel task–Program Executability Prediction (PrEx)–that asks models to predict whether a program is semantically valid or invalid (and, if invalid, which formal rule it violates) given the program’s syntax and operational semantics. Because PrEx requires both valid and invalid programs, we build a dataset with systematically generated invalid transformations derived from valid programs. We evaluate open-source coding LLMs under two semantic formalisms and two semantic shifts across Human-Written, LLM-Translated, and Fuzzer-Generated program splits. Our findings show that LLMs lean on pre-training priors rather than systematically applying the given rules, performing especially poorly on modified semantics and degrading further as program complexity increases. PrEx is available at https://github.com/EngineeringSoftware/prex.
Article Search
Article: splashws26lmplmain-p42-p (type: Full Paper) doi:10.1145/3843750.3843842
LMPL '26: "Predicting Program Exit Code ..."
Predicting Program Exit Code with LLMs and Programming Language Semantics
Lara Marinov, Aditya Thimmaiah, Jayanth Srinivasa, Junyi Jessy Li, and Milos Gligoric
(University of Texas at Austin, USA; Cisco, USA)
Large language models (LLMs) have shown proficiency in various software engineering tasks, such as code generation and translation. However, a key limitation in their performance may be their (lack of) understanding of programming-language semantics. Even when explicit semantics are given, it remains unclear whether LLMs apply those rules or lean on priors learned during pre-training instead. We study if LLMs lean on priors or given semantics with a novel task–Program Executability Prediction (PrEx)–that asks models to predict whether a program is semantically valid or invalid (and, if invalid, which formal rule it violates) given the program’s syntax and operational semantics. Because PrEx requires both valid and invalid programs, we build a dataset with systematically generated invalid transformations derived from valid programs. We evaluate open-source coding LLMs under two semantic formalisms and two semantic shifts across Human-Written, LLM-Translated, and Fuzzer-Generated program splits. Our findings show that LLMs lean on pre-training priors rather than systematically applying the given rules, performing especially poorly on modified semantics and degrading further as program complexity increases. PrEx is available at https://github.com/EngineeringSoftware/prex.
Article Search
Article: splashws26lmplmain-p42-p (type: Full Paper) doi:10.1145/3843750.3843842
LMPL '26: "FlowCheck: Helping End-Users ..."
FlowCheck: Helping End-Users Specify and Verify Intent in Vibe-Coded Web Apps
Reya Vir, Lydia Chilton, Zhuo Zhang, and Eugene Wu
(Columbia University, USA)
Vibe-coded applications often contain silent behavioral failures in which the interface appears functional even though user-visible information does not flow to the expected state or output. We introduce FlowCheck, a constraint language to specify these user-visible information flows directly through the application interface, where constraints can also be displayed and inspected without reading code, and are structured enough for reliable LLM generation. FlowCheck translates the constraints into deterministic CodeQL analyses, and we evaluate it across four applications generated via Claude Code, and compare with three coding models as bug-finding baselines. We find that FlowCheck correctly translates and flags all 30 of our injected constraint violations with no false positives. In contrast, frontier models (Claude Opus 4.7, DeepSeek V3, and Gemini Pro) showed significantly lower accuracy when prompted to find bugs in the same code, with none achieving full accuracy. This approach lets vibe coders state intent in terms of the interface they understand, and checks it deterministically against the code they do not.
Article Search
Article: splashws26lmplmain-p81-p (type: Full Paper) doi:10.1145/3843750.3843844
LMPL '26: "Invariant Pretraining for ..."
Invariant Pretraining for Robust Code Representations
Yifeng He, Yundi Xu, Christopher Castro Gaw Gonzalo, Zili Wang, and Hao Chen
(University of California at Davis, USA; University of Hong Kong, China)
Encoder-based code representation models remain widely used for discriminative tasks such as clone detection and code classification because their small size and low inference cost matter. Yet on invariant programs, semantically equivalent code with different syntax, their representations degrade substantially despite unchanged behavior. We measure this gap across four encoder baselines, two tasks, and four datasets, then test a minimal code-only continued pretraining recipe that consistently recovers part of it. Invariant pretraining (InvPT) applies semantics-preserving transformations and combines masked language modeling with multi-positive supervised contrastive learning. All augmentations of a source function are positives: self-contrast pairs use the same code with different masks, whereas invariant-contrast pairs use transformed code, providing varied difficulty without paired natural-language data. Across all model-dataset comparisons, InvPT improves robustness on transformed test sets by a median of 8.1 percentage points for clone detection (up to 11.0) and 3.6 points for code classification (up to 19.2), while matching or improving standard performance. Ablations identify multi-positive invariant contrast as the main source of these gains. Because the test transformations are composed from the pretraining operator family, the results establish invariance only to that family, not robustness in general. Thus, the contribution is not a new objective but a measurement of where encoder robustness breaks and how much of it a simple code-only recipe recovers.
Article Search
Article: splashws26lmplmain-p19-p (type: Full Paper) doi:10.1145/3843750.3843840
LMPL '26: "FlowCheck: Helping End-Users ..."
FlowCheck: Helping End-Users Specify and Verify Intent in Vibe-Coded Web Apps
Reya Vir, Lydia Chilton, Zhuo Zhang, and Eugene Wu
(Columbia University, USA)
Vibe-coded applications often contain silent behavioral failures in which the interface appears functional even though user-visible information does not flow to the expected state or output. We introduce FlowCheck, a constraint language to specify these user-visible information flows directly through the application interface, where constraints can also be displayed and inspected without reading code, and are structured enough for reliable LLM generation. FlowCheck translates the constraints into deterministic CodeQL analyses, and we evaluate it across four applications generated via Claude Code, and compare with three coding models as bug-finding baselines. We find that FlowCheck correctly translates and flags all 30 of our injected constraint violations with no false positives. In contrast, frontier models (Claude Opus 4.7, DeepSeek V3, and Gemini Pro) showed significantly lower accuracy when prompted to find bugs in the same code, with none achieving full accuracy. This approach lets vibe coders state intent in terms of the interface they understand, and checks it deterministically against the code they do not.
Article Search
Article: splashws26lmplmain-p81-p (type: Full Paper) doi:10.1145/3843750.3843844
LMPL '26: "Invariant Pretraining for ..."
Invariant Pretraining for Robust Code Representations
Yifeng He, Yundi Xu, Christopher Castro Gaw Gonzalo, Zili Wang, and Hao Chen
(University of California at Davis, USA; University of Hong Kong, China)
Encoder-based code representation models remain widely used for discriminative tasks such as clone detection and code classification because their small size and low inference cost matter. Yet on invariant programs, semantically equivalent code with different syntax, their representations degrade substantially despite unchanged behavior. We measure this gap across four encoder baselines, two tasks, and four datasets, then test a minimal code-only continued pretraining recipe that consistently recovers part of it. Invariant pretraining (InvPT) applies semantics-preserving transformations and combines masked language modeling with multi-positive supervised contrastive learning. All augmentations of a source function are positives: self-contrast pairs use the same code with different masks, whereas invariant-contrast pairs use transformed code, providing varied difficulty without paired natural-language data. Across all model-dataset comparisons, InvPT improves robustness on transformed test sets by a median of 8.1 percentage points for clone detection (up to 11.0) and 3.6 points for code classification (up to 19.2), while matching or improving standard performance. Ablations identify multi-positive invariant contrast as the main source of these gains. Because the test transformations are composed from the pretraining operator family, the results establish invariance only to that family, not robustness in general. Thus, the contribution is not a new objective but a measurement of where encoder robustness breaks and how much of it a simple code-only recipe recovers.
Article Search
Article: splashws26lmplmain-p19-p (type: Full Paper) doi:10.1145/3843750.3843840
LMPL '26: "Detecting Database Migration ..."
Detecting Database Migration Non-equivalence through LLM-Assisted Mutation-Score-Guided Testing
Yuhao Zhang, Sayan Ghosh, Amit Kachroo, and Jun Huan
(AWS AI Labs, USA)
Database migration involves translating schemas, data, and queries from one database system to another. Query translation, a key step in this process, is widely practiced but error-prone: subtle semantic differences between SQL dialects can cause silent data corruption and incorrect query results. As migrations increasingly rely on LLMs, the translated queries lack equivalence validations. Existing formal verification tools cannot handle the full range of features in SQL dialects, and differential testing provides no confidence measure when all tests pass. We propose LAMT, an LLM-assisted mutation-score-guided testing approach that addresses both problems. Given a pair of pre-migration and post-migration SQL queries, LAMT operates as an adversarial loop: an LLM table generator produces test data aimed at killing query mutants, i.e., exposing behavioral differences, while an LLM query generator produces mutants that attempt to evade detection by the current test suite. The resulting mutation score is then calibrated into a confidence threshold that estimates the risk of undetected non-equivalent pairs. We evaluate LAMT on 1,035 cross-dialect migration pairs spanning three dialect pairs. Query-aware table generation reduces the average false negative rate from 58.3% to 20.0% with 4 more generated tables. The mutation score calibration provides directional confidence estimates, though a gap between estimated and true FNR remains, particularly for weaker models.
Article Search
Article: splashws26lmplmain-p85-p (type: Full Paper) doi:10.1145/3843750.3843845
LMPL '26: "FlowCheck: Helping End-Users ..."
FlowCheck: Helping End-Users Specify and Verify Intent in Vibe-Coded Web Apps
Reya Vir, Lydia Chilton, Zhuo Zhang, and Eugene Wu
(Columbia University, USA)
Vibe-coded applications often contain silent behavioral failures in which the interface appears functional even though user-visible information does not flow to the expected state or output. We introduce FlowCheck, a constraint language to specify these user-visible information flows directly through the application interface, where constraints can also be displayed and inspected without reading code, and are structured enough for reliable LLM generation. FlowCheck translates the constraints into deterministic CodeQL analyses, and we evaluate it across four applications generated via Claude Code, and compare with three coding models as bug-finding baselines. We find that FlowCheck correctly translates and flags all 30 of our injected constraint violations with no false positives. In contrast, frontier models (Claude Opus 4.7, DeepSeek V3, and Gemini Pro) showed significantly lower accuracy when prompted to find bugs in the same code, with none achieving full accuracy. This approach lets vibe coders state intent in terms of the interface they understand, and checks it deterministically against the code they do not.
Article Search
Article: splashws26lmplmain-p81-p (type: Full Paper) doi:10.1145/3843750.3843844