MET 2024 – Proceedings

AI-Driven Metamorphic Testing for Autonomous Systems (Keynote)
Arnaud Gotlieb
(Simula Research Laboratory, Norway)
Autonomous systems such as automated driving systems, autonomous ships or industrial collaborative robots are data-intensive software systems that embed self-adaptive and self-reasoning capabilities. Even though increased autonomy is highly desirable, validation challenges come with it as autonomous systems are considered non-testable. Indeed, their exact behaviour is hardly predictable as they depend on datasets, trained models and observable execution environments. Fortunately, Metamorphic Testing techniques have been developed for a quarter of a century to support test engineers in validating safety- and business-critical software systems. My talk will review the main principles of Metamorphic Testing and detail two specific approaches where reinforcement learning and automated AI planning are used to address the testability problem of selected non-testable autonomous systems. A few lessons learned will allow me to draw some research perspectives for the progress of Metamorphic Testing.

Publisher's Version

Papers

Using Metamorphic Relations to Improve Accuracy and Robustness of Deep Neural Networks
Kun Qiu, Yu Zhou, and Pak-Lok Poon
(Hefei University of Technology, China; Central Queensland University, Australia)
When applying metamorphic testing to a deep neural network (DNN), the DNN could have an "acceptable" level of accuracy but it performs poorly against some metamorphic relations (MRs). Such a DNN is considered not robust against these MRs. Improving both accuracy and robustness of a DNN is non-trivial because improving one aspect may adversely affect the other aspect. To alleviate this trade-off problem, we proposed a regularization-based method, in which an optimization function is designed to balance a DNN's accuracy and robustness. Then, we designed a reinforcement-learning-based algorithm to optimize this function. We tested our training method with two datasets (SVHN and CIFAR10), and each dataset with two DNN models. When comparing ours with the other six benchmark methods, we found the DNNs trained with our method have a better balance between accuracy and robustness.

Publisher's Version

Using Category Partition to Detect Metamorphic Relations
Saba Pedram and Yvan Labiche
(Carleton University, Canada)
The Category Partition (CP) functional testing method has proven to be useful in various contexts. It begins by identifying parameters and environment conditions on the basis of the function's behaviour. The characteristics/categories of these parameters/environment conditions are identified and partitioned into choices. The choices of a category are mutually exclusive and can be based on input partitioning and boundary value analysis. Thereafter, the choices are combined on the basis of a selection criterion to form test frames. Once input values satisfying the conditions of a test frame's choices are identified, one is equipped with a test case. This paper suggests and demonstrates that those test frames, once equipped with characterizations of output values, i.e., with categories and choices for outputs, can be considered Metamorphic Relations to be used in Metamorphic Testing.

Publisher's Version

Metamorphic Testing of a Steer-by-Wire System: An Intercultural Students-as-Partners Collaboration Experience
Yifan Zhang, Dave Towey, Matthew Pike, Rui Qiu, Axel Tan Jaya, Sze Huey, Xinyi Zhang, and Yuan Wu
(University of Nottingham Ningbo China, China)
This paper explores the educational and practical impacts of integrating metamorphic testing (MT) into a software engineering project conducted by an intercultural group of students. The students designed a Steer-by-Wire (SBW) system to control the steering of a model vehicle and tested using a hybrid approach that combined unit testing and MT. Four metamorphic relations (MRs) were generated and two significant violations were encountered during the testing phase. The first violation, related to steering angle consistency reported by the system, as a case of metamorphic exploration (ME), revealed a common coding mistake where the system failed to maintain consistent steering angles for equivalent inputs in opposite directions, illustrating how ME can enhance comprehension of the system and the testing process itself. It not only deepened the testers' understanding of the integration between software and mechanical systems but also represented valuable insights for others engaged in similar tasks. The second MR violation revealed issues with interruptions and delays when the system switched between manual and automated control modes, demonstrating MT's effectiveness in identifying defects and highlighting MT's importance in real-world software development scenarios. Additionally, the project examined the effectiveness of aligning MT roles to team members based on their Myers-Briggs Type Indicator (MBTI) personalities, suggesting that such alignments can enhance team dynamics and overall project efficiency. This study provides insights into the benefits of using MT in educational settings, the implications of personality-based task assignments, and the enhancement of software reliability and team performance in an intercultural context. The findings of this research reinforce the value of MT in software engineering education and support for the integration of psychological analysis in managing complex projects.

Publisher's Version

Metamorphic Testing of Image Processing Applications: A General Framework and Optimization Strategies
Chang-Ai Sun, Jiayu Xing, Xiaobei Li, Xiaoyi Zhang, and An Fu
(University of Science and Technology Beijing, China)
Metamorphic testing (MT) is widely adopted for testing image processing applications. Although a variety of metamorphic relations (MRs) have been proposed, using all of them for testing will cost a large amount of computational resources. In addition, complex transformation operations are not well supported when generating follow-up test images based on MRs. To overcome these limitations, this study proposes a general MT framework for image processing applications, which employs CycleGAN to generate images that are very close to the realistic scenarios and leverages MRs for various categories of image processing applications. Two optimization strategies called EquivalentMR and SSampling are further proposed to reduce MRs and test images, respectively. A prototype tool called MT4I was developed. The experimental results showed that the proposed framework was capable of effectively testing various categories of image processing applications, while optimization strategies can reduce the amounts of MRs and test images without significantly jeopardizing the fault detection effectiveness.

Publisher's Version

Evaluating Human Trajectory Prediction with Metamorphic Testing
Helge Spieker, Nassim Belmecheri, Arnaud Gotlieb, and Nadjib Lazaar
(Simula Research Laboratory, Norway; LIRMM, France; University of Montpellier, France; CNRS, France)
The prediction of human trajectories is important for planning in autonomous systems that act in the real world, e.g. automated driving or mobile robots. Human trajectory prediction is a noisy process, and no prediction does precisely match any future trajectory. It is therefore approached as a stochastic problem, where the goal is to minimise the error between the true and the predicted trajectory. In this work, we explore the application of metamorphic testing for human trajectory prediction. Metamorphic testing is designed to handle unclear or missing test oracles. It is well-designed for human trajectory prediction, where there is no clear criterion of correct or incorrect human behaviour. Metamorphic relations rely on transformations over source test cases and exploit invariants. A setting well-designed for human trajectory prediction where there are many symmetries of expected human behaviour under variations of the input, e.g. mirroring and rescaling of the input data. We discuss how metamorphic testing can be applied to stochastic human trajectory prediction and introduce the Wasserstein Violation Criterion to statistically assess whether a follow-up test case violates a label-preserving metamorphic relation.

Publisher's Version

MET 2024 – Proceedings

9th ACM International Workshop on Metamorphic Testing (MET 2024)

Frontmatter

Keynote

Papers