TORACLE '21: "Using Machine Learning to ..."
Using Machine Learning to Generate Test Oracles: A Systematic Literature Review
Afonso Fontes and Gregory Gay
(Chalmers University of Technology, Sweden; University of Gothenburg, Sweden)
Machine learning may enable the automated generation of test oracles. We have characterized emerging research in this area through a systematic literature review examining oracle types, researcher goals, the ML techniques applied, how the generation process was assessed, and the open research challenges in this emerging field.
Based on a sample of 22 relevant studies, we observed that ML algorithms generated test verdict, metamorphic relation, and---most commonly---expected output oracles. Almost all studies employ a supervised or semi-supervised approach, trained on labeled system executions or code metadata---including neural networks, support vector machines, adaptive boosting, and decision trees. Oracles are evaluated using the mutation score, correct classifications, accuracy, and ROC. Work-to-date show great promise, but there are significant open challenges regarding the requirements imposed on training data, the complexity of modeled functions, the ML algorithms employed---and how they are applied---the benchmarks used by researchers, and replicability of the studies. We hope that our findings will serve as a roadmap and inspiration for researchers in this field.
@InProceedings{TORACLE21p1,
author = {Afonso Fontes and Gregory Gay},
title = {Using Machine Learning to Generate Test Oracles: A Systematic Literature Review},
booktitle = {Proc.\ TORACLE},
publisher = {ACM},
pages = {1--10},
doi = {10.1145/3472675.3473974},
year = {2021},
}
Publisher's Version
TORACLE '21: "Using Machine Learning to ..."
Using Machine Learning to Generate Test Oracles: A Systematic Literature Review
Afonso Fontes and Gregory Gay
(Chalmers University of Technology, Sweden; University of Gothenburg, Sweden)
Machine learning may enable the automated generation of test oracles. We have characterized emerging research in this area through a systematic literature review examining oracle types, researcher goals, the ML techniques applied, how the generation process was assessed, and the open research challenges in this emerging field.
Based on a sample of 22 relevant studies, we observed that ML algorithms generated test verdict, metamorphic relation, and---most commonly---expected output oracles. Almost all studies employ a supervised or semi-supervised approach, trained on labeled system executions or code metadata---including neural networks, support vector machines, adaptive boosting, and decision trees. Oracles are evaluated using the mutation score, correct classifications, accuracy, and ROC. Work-to-date show great promise, but there are significant open challenges regarding the requirements imposed on training data, the complexity of modeled functions, the ML algorithms employed---and how they are applied---the benchmarks used by researchers, and replicability of the studies. We hope that our findings will serve as a roadmap and inspiration for researchers in this field.
@InProceedings{TORACLE21p1,
author = {Afonso Fontes and Gregory Gay},
title = {Using Machine Learning to Generate Test Oracles: A Systematic Literature Review},
booktitle = {Proc.\ TORACLE},
publisher = {ACM},
pages = {1--10},
doi = {10.1145/3472675.3473974},
year = {2021},
}
Publisher's Version