Powered by
Conference Publishing Consulting

Sixth International Workshop on Automation of Software Test (AST 2011), May 23-24, 2011, Waikiki, Honolulu, HI, USA

AST 2011 – Proceedings

Contents - Abstracts - Authors

Sixth International Workshop on Automation of Software Test (AST 2011)


Title Page

Workshop on Automation of Software Test (AST2011) Foreword As proud co-chairs of the 6th workshop on Automation of Software Test, we would like to take this opportunity to welcome you to AST2011 and to thank all participants for taking the time out from their busy lives and the work in their home countries to make the effort to attend this workshop. The theme of this year’s workshop, aligned with the ICSE 2011 conference theme, is Relating Software Design to Test Automation. The theme covers several prominent perspectives both in Software Design and Automation of Software Test practice. The broad scope of AST is reflected in the wide range of topics covered by the 36 submissions that we received. Of these, with the 32 members of the AST Program Committee from both academic and industrial research labs, we selected 15 regular research papers that are grouped into six sessions, representing a diversity of test automation related topics. The six sessions are: • Model-Based Testing • Testing for Web and Service-Based Systems • Testing Strategies I and II • Application of Testing I and II The schedule of the two workshop days will be intensive yet productive. Besides the six regular paper sessions, we will have two poster sessions covering promising ongoing research work in test automation. The program of each day will start from a prominent keynote speech, one from academy and the other from industry. A multi-day charette on the key near-term challenges in relating software architecture and automated software test will be the highlight of this workshop. On the first day, the participants will identify challenges related to software architecture support for testing and divide into teams of a half dozen people. Each team will take up one of the related questions and create a feasible near-term research pathway to effectively address it. On the second day, each team will finalize a presentation on their challenge and pathway, and will then present their ideas to the workshop as a whole. We hope that this workshop will be an enjoyable and productive opportunity for you to meet and discuss various testing automation issues with your counterparts from other nations and other industrial segments. We would like to thank all the many people who contributed to make this workshop a reality, including the AST 2011 Steering and Program Committees, the external reviewers, the ICSE 2011 Workshop Chairs Matthew Dwyer and Wolfgang Emmerich and all the authors and participants. Again welcome to AST2011, and enjoy your time here. Antonia, Howard and J. Jenny

Model-Based Testing

Metamorphic Testing of a Monte Carlo Modeling Program
Junhua Ding, Tong Wu, Dianxiang Xu, Jun Q. Lu, and Xin-Hua Hu
(East Carolina University, USA; Dakota State University, USA)
Photon propagation in biological tissue can be equivalently modeled with Monte Carlo simulations numerically or by the Radiative Transfer Equation (RTE) analytically. However, testing of a Monte Carlo program modeling photon propagation in biological tissue is difficult due to the unknown character of the test oracles. Although approaches based on Beer-Lambert law, van de Hulst’s table or RTE can be used for testing the Monte Carlo modeling program, these approaches are only applied to the program that is designed for homogeneous media. A rigorous way for testing the Monte Carlo modeling program for heterogeneous media is needed. In this paper, we investigate the effectiveness of the metamorphic testing approach to test a Monte Carlo modeling program for heterogeneous media. In addition, the metamorphic testing is extended with the evaluation of the adequacy of testing coverage criteria to measure the quality of the metamorphic testing, to guide the creation of metamorphic relations, to generate testing inputs, and to investigate the found exceptions. The effectiveness of the approach has been demonstrated through testing a Monte Carlo modeling program we developed for simulating photon propagation in human skins.
Article Search
Abstracting Timing Information in UML State Charts via Temporal Ordering and LOTOS
Valentin Chimisliu and Franz Wotawa
(TU Graz, Austria)
As testing of software systems becomes more and more important and expensive, there is a trend to automate as much as possible of this task. This article is intended as an attempt to breach the gap between academic model-based testing tools and their usage in industry. This is done by allowing the specification of a system in a widely accepted industry notation (UML state charts) and via a behind the scene transformation providing a formal representation of the system using the formal language LOTOS. As a byproduct of the transformation a formal semantics of UML state charts is given. An interesting class of software systems well suited for the application are distributed timed control oriented systems. As LOTOS contains no timing constructs, the timing information in the system is automatically abstracted by preserving the execution order of the timeout transitions.
Article Search
Scalable Graph Analyzing Approach for Software Fault-Localization
Mojtaba Vahidi-Asl and Saeed Parsa
(Iran University of Science and Technology, Iran)
In this paper, a new approach for analyzing program behavioral graphs to detect fault relevant paths is presented. The existing graph mining approaches for bug localization merely detect discriminative sub-graphs between failing and passing runs. However, they are not applicable when the context of a failure is not appeared in a discriminative pattern. In our proposed method, the suspicious transitions are identified by contrasting nearest neighbor failing and passing dynamic behavioral graphs. For finding similar failing and passing graphs, we first convert the graphs into adequate vectors. Then, a combination of Jacard-Cosine similarity measures is applied to identify the nearest graphs. The new scoring formula takes advantage of null hypothesis testing for ranking weighted transitions. The main advantage of the proposed technique is its scalability which makes it work on large and complex programs with huge number of predicates. Another main capability of our approach is providing the faulty paths constructed from fault suspicious transitions. Considering the weighted execution graphs in the analysis enables us to find those types of bugs which reveal themselves in specific number of transitions between two particular predicates. The experimental results on Siemens test suite and Space program manifest the effectiveness of the proposed method on weighted execution graphs for locating bugs in comparison with other methods.
Article Search

Testing for Web and Service-Based Systems

Design of Intelligent Agents for Collaborative Testing of Service-Based Systems
Xiaoying Bai, Bin Chen, Bo Ma, and Yunzhan Gong
(Tsinghua University, China; Beijing University of Posts and Telecommunications, China)
Testing on services-based systems faces the challenges of dynamic collaboration. Services are distributed software that can be bound to establish collaborations on-demand. To verify and validate the services, testing needs to react automatically in a coordinated approach. Software agents, which are characterized by persistence, autonomy, social ability and reactivity, are thus introduced to facilitate test deployment, execution, collaboration, and run-time decision making. This paper proposes a design of test agent model, including agents' knowledge, events, actions and interpreter. The knowledge represents the detected environment status, such as test results and changes in services under test. The action models testing behavior such as test configuration, test deployment and test schedule. The Interpreter defines the rules to select actions or parameters on certain events and conditions. In this way, given a set of knowledge at a certain time, a test agent dynamically adjusts its behavior according to its pre-defined rules and strategies. Case studies and experiments are exercised to apply the generic agent design to specific testing tasks such as performance testing and coverage-based testing.
Article Search
A Comparative Evaluation of State-of-the-Art Web Service Composition Testing Approaches
Hazlifah Mohd Rusli, Mazidah Puteh, Suhaimi Ibrahim, and Sayed Gholam Hassan Tabatabaei
(Universiti Teknologi MARA, Malaysia; Universiti Teknologi Malaysia, Malaysia)
More and more Web based systems are being developed by composing other single or even composite services. This is due to the fact that not all available services are able to satisfy the needs of a user. The process of composing Web services involves discovering the appropriate services, selecting the best services, combining those services together, and finally executing them. Although much research efforts have been dedicated to the discovery, selection, and composition of services, the process of testing the Web service composition has not been given the same attention. This paper discusses the importance of Web services composition testing, provides a classification of the most prominent approaches in that area, presents several criteria for comparison of those approaches, and conducts a comparative evaluation of the approaches. The results of the paper give an essential perspective to do research work on Web services composition testing.
Article Search

Test Strategies I

A Comfortable TestPlayer for Analyzing Statistical Usage Testing Strategies
Winfried Dulz
(University of Erlangen-Nuremberg, Germany)
We will first give a brief introduction to a versatile modeling environment that allows the early validation of system specifications by a test-driven agile simulation approach. The main focus of our paper is on providing techniques for automated test case generation relying on statistical usage models. Based on the open source software R for statistical computing and graphics the easy to handle test case generation and analyzing Testplayer tool was developped. Starting from customer-specific graphical Markov chain usage models test cases are automatically generated and visualized by highlighting selected nodes and arcs in the model. In addition, various metrics and corresponding diagrams offer analytical techniques to assess the quality of the derived test suite.
Article Search
Automatically Testing Interactive Multimodal Systems Using Task Trees and Fusion Models
Laya Madani and Ioannis Parissis
(University Al Baath, Syria and Laboratoire d’Informatique de Grenoble, France; University of Grenoble and Grenoble INP - LCIS, France)
Multimodal systems support communication with the user through different modalities such as voice and gesture. In such systems, modalities may be used sequentially or concurrently, and independently or combined synergistically. Their use is characterized by four properties, called CARE (Complementarity, Assignment, Redundancy and Equivalence). This paper presents a method for generating automatically test data for multimodal systems, based on two models: a model describing the multimodal aspects and a task tree model, both supporting operational profile specification.
Article Search
Using Conditional Mutation to Increase the Efficiency of Mutation Analysis
René Just, Gregory M. Kapfhammer, and Franz Schweiggert
(Ulm University, Germany; Allegheny College, USA)
Assessing testing strategies and test sets is a crucial part of software testing. Mutation analysis is, among other approaches, a suitable technique for this purpose. However, compared with other methods it is rather time-consuming and applying mutation analysis to large software systems is still problematic. This paper presents a versatile approach, called conditional mutation, which increases the efficiency of mutation analysis. This new method significantly reduces the time overhead for generating and executing the mutants. Results are reported for eight investigated programs up to 373,000 lines of code and 406,000 generated mutants. Furthermore, conditional mutation has been integrated into the Java 6 Standard Edition compiler. Thus, it is widely applicable and not limited to a certain testing tool or framework.
Article Search

Test Strategies II

Better Predicate Testing
Gary Kaminski, Paul Ammann, and Jeff Offutt
(George Mason University, USA)
Mutation testing is widely recognized as being extremely powerful, but is considered difficult to automate enough for practical use. This paper theoretically addresses two possible reasons for this: the generation of redundant mutants and the lack of integration of mutation analysis with other test criteria. By addressing these two issues, this paper brings an important mutation operator, relational-operator-replacement (ROR), closer to practical use. First, we develop fault hierarchies for the six relational operators, each of which generates seven mutants per clause. These hierarchies show that, for any given clause, only three mutants are necessary. This theoretical result can be integrated easily into mutation analysis tools, thereby eliminating generation of 57% of the ROR mutants. Second, we show how to bring the power of the ROR operator to the widely used Multiple Condition-Decision Coverage (MCDC) test criterion. This theoretical result includes an algorithm to transform any MCDC-adequate test set into a test set that also satisfies RORG, a new version of ROR appropriate for the MCDC context. The transformation does not use traditional mutation analysis, so can easily be integrated into existing MCDC tools and processes.
Article Search
Hazard-based Selection of Test Cases
Mario Gleirscher
(TU München, Germany)
It is a challenge to engineer and select admissible test cases. This work approaches selection based on hazardous states through integrated modelling of the environment and the system, and by integration of requirements engineering activities into the tasks of system test engineering. The elaboration of test models and test case specifications focuses automation in system verification in order to detect hazard-oriented failures as well as in requirements validation in order to disclose potential defects in specification artefacts.
Article Search

Applications of Testing I

Automated Testing of Industrial Control Devices: The Delphi Database
Nate Kube, Kevin Yoo, and Daniel Hoffman
(Wurldtech Security Technologies, Canada; University of Victoria, Canada)
Delphi is a database designed to centralize and distribute current industrial automation vulnerability information. Over the past two years, with the aid of many of the world’s largest equipment vendors and operators, we have populated this database through extensive testing of industrial control devices. Delphi stores over 500 vulnerabilities on 31 popular distributed control system and safety instrumented system controllers. There are two primary reasons why Delphi data is useful: to distribute vulnerability data and to provide mitigation strategies. With detailed knowledge of which vulnerabilities are present, vendors can produce more robust devices and operators can construct business cases and calculate return-on-investment when considering investments in security measures. Furthermore, known vulnerabilities can be mitigated without applying device patches and shutting down plant operations.
Article Search
Automating GUI Testing for Android Applications
Cuixiong Hu and Iulian Neamtiu
(UC Riverside, USA)
Users increasingly rely on mobile applications for computational needs. Google Android is a popular mobile platform, hence the reliability of Android applications is becoming increasingly important. Many Android correctness issues, however, fall outside the scope of traditional verification techniques, as they are due to the novelty of the platform and its GUI-oriented application construction paradigm. In this paper we present an approach for automating the testing process for Android applications, with a focus on GUI bugs. We first conduct a bug mining study to understand the nature and frequency of bugs affecting Android applications; our study finds that GUI bugs are quite numerous. Next, we present techniques for detecting GUI bugs by automatic generation of test cases, feeding the application random events, instrumenting the VM, producing log/trace files and analyzing them post-run. We show how these techniques helped to re-discover existing bugs and find new bugs, and how they could be used to prevent certain bug categories. We believe our study and techniques have the potential to help developers increase the quality of Android applications.
Article Search

Applications of Testing II

Selection and Execution of User Level Test Cases for Energy Cost Evaluation of Smartphones
Rajesh Palit, Renuka Arya, Kshirasagar Naik, and Ajit Singh
(University of Waterloo, Canada)
Smartphones are emerging as a preferred communication device of users. In this paper, we provide a methodology to select user level test cases for performing energy cost evaluation of smartphone applications. We define the concept of a user level test case for smartphones and show that, due to configuration settings, there exist millions of such test cases. Next, we discuss a test selection technique to reduce the number of test cases. We apply the technique to five different smartphones and evaluate their energy costs for running common network related applications. We have developed a test bench to execute those test cases for real applications on smartphones and measure their actual energy costs. This work provides a framework for researchers and developers to conduct experiments for measuring the energy cost of applications on smartphones.
Article Search
Testing an Optimising Compiler by Generating Random Lambda Terms
Michał H. Pałka, Koen Claessen, Alejandro Russo, and John Hughes
(Chalmers University of Technology, Sweden; Quviq AB, Sweden)
This paper considers random testing of a compiler, using randomly generated programs as inputs, and comparing their behaviour with and without optimisation. Since the generated programs must compile, then we need to take into account syntax, scope rules, and type checking during our random generation. Doing so, while attaining a good distribution of test data, proves surprisingly subtle; the main contribution of this paper is a workable solution to this problem. We used it to generate typed functions on lists, which we compiled using the Glasgow Haskell compiler, a mature production quality Haskell compiler. After around 20,000 tests we triggered an optimiser failure, and automatically simplified it to a program with just a few constructs.
Article Search
Model-Driven Design and Validation of Embedded Software
Giuseppe Di Guglielmo, Masahiro Fujita, Luigi Di Guglielmo, Franco Fummi, Graziano Pravadelli, Cristina Marconcini, and Andreas Foltinek
(The University of Tokyo, Japan; University of Verona, Italy; STM Products srl, Italy; IMACS GmbH, Germany)

Article Search

Poster Session 1

Test Data to Reduce the Complexity of Unit Test Automation
Guy Collins Ndem, Abbas Tahir, Andreas Ulrich, and Helmut Goetz
(Siemens AG, Germany)
In order to automatically test large and complex software systems, a well defined test process and a huge amount of test data are needed. When it comes to test automation, the quality of test data always plays a significant role by reducing the cost and the time required for the test activities. In this work, we follow the principle of "Quality of the test cases comes before Quantity" to show how the right selection of test data helps to address the goal of qualitative testing while optimizing the test effort needed
Article Search
A Framework for Automatic Functional Testing Based on Formal Specifications
Shaoying Liu and Shin Nakajima
(Hosei University, Japan; NII, Japan)
This paper describes a framework for automatic functional testing based on formal specifications. The framework is intended to expose all of the issues that must be addressed in automatic testing, to provide solutions for some of the major challenges, and to serve as a foundation for developing tool support in the future.
Article Search

Poster Session 2

Towards Automated Testing of Web Service Choreographies
Felipe M. Besson, Pedro M. B. Leal, Fabio Kon, Alfredo Goldman, and Dejan Milojicic
(University of São Paulo, Brazil; Hewlett Packard Laboratories, USA)
Web service choreographies have been proposed as a decentralized scalable way of composing services in a SOA environment. In spite of all the benefits of choreographies, the decentralized flow of information, the parallelism, and multiple party communication restrict the automated testing of choreographies at design and runtime. The goal of our research is to adapt the automated testing techniques used by the Agile Software Development community to the SOA context. To achieve that, we seek to develop software tools and a methodology to enable test-driven development of Choreographies. In this paper, we present our first step in that direction, a software prototype composed of ad hoc automated test case scripts for testing a web service choreography.
Article Search
monadWS: A Monad-Based Testing Tool for Web Services
Yingzhou Zhang, Wei Fu, and Changhai Nie
(Nanjing University of Posts and Telecommunications, China; Nanjing University, China)
Testing Web Services (WS) is very important because it could guarantee a high degree of service quality and reliability. Automatic WS testing tools enable testers to complete testing in a shorter time, and they make it easy to repeat tests after each modification to a service. In this paper, we present monadWS, a prototype tool of monad-based automated testing for WS, which enables concrete description, autogeneration and execution for WS test cases with the design of service testing monads. The results show that monadWS is feasible and effective for testing WS.
Article Search
Towards Automated Oracles for GUI Input Validation
Gabriel L. Zenarosa and Regis J. Leonard
(University of Pittsburgh, USA)
Testing input validation in web applications from specifications is a challenging and laborious process. GUI testing tools---with their record-and-playback and data-driven capabilities---ease the pains of testing through automation. Out-of-the-box, however, these tools have some scaling limitations as setup costs are incurred for every distinct web application to test. In environments where a line of many web applications are regularly created for various customers and purposes, scaling the test automation to span the entire product line is extremely valuable. In this paper, we report on our experience in generalizing the automatic specification-based testing of input validation in a line of web applications. Our approach is based on a nonstandard use of a GUI testing tool enabled by adjustments to coding standards and the requirements specification writing process.
Article Search

proc time: 0.06