Powered by
Sixth International Workshop on Automation of Software Test (AST 2011),
May 23-24, 2011,
Waikiki, Honolulu, HI, USA
Sixth International Workshop on Automation of Software Test (AST 2011)
Preface
Foreword
Workshop on Automation of Software Test (AST2011) Foreword
As proud co-chairs of the 6th workshop on Automation of Software Test, we would like to take this opportunity to welcome you to AST2011 and to thank all participants for taking the time out from their busy lives and the work in their home countries to make the effort to attend this workshop.
The theme of this year’s workshop, aligned with the ICSE 2011 conference theme, is Relating Software Design to Test Automation. The theme covers several prominent perspectives both in Software Design and Automation of Software Test practice. The broad scope of AST is reflected in the wide range of topics covered by the 36 submissions that we received. Of these, with the 32 members of the AST Program Committee from both academic and industrial research labs, we selected 15 regular research papers that are grouped into six sessions, representing a diversity of test automation related topics. The six sessions are:
• Model-Based Testing
• Testing for Web and Service-Based Systems
• Testing Strategies I and II
• Application of Testing I and II
The schedule of the two workshop days will be intensive yet productive. Besides the six regular paper sessions, we will have two poster sessions covering promising ongoing research work in test automation. The program of each day will start from a prominent keynote speech, one from academy and the other from industry. A multi-day charette on the key near-term challenges in relating software architecture and automated software test will be the highlight of this workshop. On the first day, the participants will identify challenges related to software architecture support for testing and divide into teams of a half dozen people. Each team will take up one of the related questions and create a feasible near-term research pathway to effectively address it. On the second day, each team will finalize a presentation on their challenge and pathway, and will then present their ideas to the workshop as a whole.
We hope that this workshop will be an enjoyable and productive opportunity for you to meet and discuss various testing automation issues with your counterparts from other nations and other industrial segments. We would like to thank all the many people who contributed to make this workshop a reality, including the AST 2011 Steering and Program Committees, the external reviewers, the ICSE 2011 Workshop Chairs Matthew Dwyer and Wolfgang Emmerich and all the authors and participants. Again welcome to AST2011, and enjoy your time here.
Antonia, Howard and J. Jenny
Model-Based Testing
Metamorphic Testing of a Monte Carlo Modeling Program
Junhua Ding and
Xin-Hua Hu
(East Carolina University, USA; Dakota State University, USA)
Photon propagation in biological tissue can be equivalently modeled with Monte Carlo simulations numerically or by the Radiative Transfer Equation (RTE) analytically. However, testing of a Monte Carlo program modeling photon propagation in biological tissue is difficult due to the unknown character of the test oracles. Although approaches based on Beer-Lambert law, van de Hulst’s table or RTE can be used for testing the Monte Carlo modeling program, these approaches are only applied to the program that is designed for homogeneous media. A rigorous way for testing the Monte Carlo modeling program for heterogeneous media is needed. In this paper, we investigate the effectiveness of the metamorphic testing approach to test a Monte Carlo modeling program for heterogeneous media. In addition, the metamorphic testing is extended with the evaluation of the adequacy of testing coverage criteria to measure the quality of the metamorphic testing, to guide the creation of metamorphic relations, to generate testing inputs, and to investigate the found exceptions. The effectiveness of the approach has been demonstrated through testing a Monte Carlo modeling program we developed for simulating photon propagation in human skins.
@InProceedings{AST11p1,
author = {Junhua Ding and Xin-Hua Hu},
title = {Metamorphic Testing of a Monte Carlo Modeling Program},
booktitle = {Proc.\ AST},
publisher = {ACM},
pages = {1--7},
doi = {},
year = {2011},
}
Abstracting Timing Information in UML State Charts via Temporal Ordering and LOTOS
Valentin Chimisliu and
Franz Wotawa
(TU Graz, Austria)
As testing of software systems becomes more and more important
and expensive, there is a trend to automate as much
as possible of this task. This article is intended as an attempt
to breach the gap between academic model-based
testing tools and their usage in industry. This is done by
allowing the specification of a system in a widely accepted
industry notation (UML state charts) and via a behind the
scene transformation providing a formal representation of
the system using the formal language LOTOS. As a byproduct
of the transformation a formal semantics of UML
state charts is given. An interesting class of software systems
well suited for the application are distributed timed
control oriented systems. As LOTOS contains no timing
constructs, the timing information in the system is automatically
abstracted by preserving the execution order of
the timeout transitions.
@InProceedings{AST11p8,
author = {Valentin Chimisliu and Franz Wotawa},
title = {Abstracting Timing Information in UML State Charts via Temporal Ordering and LOTOS},
booktitle = {Proc.\ AST},
publisher = {ACM},
pages = {8--14},
doi = {},
year = {2011},
}
Scalable Graph Analyzing Approach for Software Fault-Localization
Mojtaba Vahidi-Asl and
Saeed Parsa
(Iran University of Science and Technology, Iran)
In this paper, a new approach for analyzing program behavioral graphs to detect fault relevant paths is presented. The existing graph mining approaches for bug localization merely detect discriminative sub-graphs between failing and passing runs. However, they are not applicable when the context of a failure is not appeared in a discriminative pattern. In our proposed method, the suspicious transitions are identified by contrasting nearest neighbor failing and passing dynamic behavioral graphs. For finding similar failing and passing graphs, we first convert the graphs into adequate vectors. Then, a combination of Jacard-Cosine similarity measures is applied to identify the nearest graphs. The new scoring formula takes advantage of null hypothesis testing for ranking weighted transitions. The main advantage of the proposed technique is its scalability which makes it work on large and complex programs with huge number of predicates. Another main capability of our approach is providing the faulty paths constructed from fault suspicious transitions. Considering the weighted execution graphs in the analysis enables us to find those types of bugs which reveal themselves in specific number of transitions between two particular predicates. The experimental results on Siemens test suite and Space program manifest the effectiveness of the proposed method on weighted execution graphs for locating bugs in comparison with other methods.
@InProceedings{AST11p15,
author = {Mojtaba Vahidi-Asl and Saeed Parsa},
title = {Scalable Graph Analyzing Approach for Software Fault-Localization},
booktitle = {Proc.\ AST},
publisher = {ACM},
pages = {15--21},
doi = {},
year = {2011},
}
Testing for Web and Service-Based Systems
Design of Intelligent Agents for Collaborative Testing of Service-Based Systems
Xiaoying Bai, Bin Chen, and
Yunzhan Gong
(Tsinghua University, China; Beijing University of Posts and Telecommunications, China)
Testing on services-based systems faces the challenges of dynamic collaboration. Services are distributed software that can be bound to establish collaborations on-demand. To verify and validate the services, testing needs to react automatically in a coordinated approach. Software agents, which are characterized by persistence, autonomy, social ability and reactivity, are thus introduced to facilitate test deployment, execution, collaboration, and run-time decision making. This paper proposes a design of test agent model, including agents' knowledge, events, actions and interpreter. The knowledge represents the detected environment status, such as test results and changes in services under test. The action models testing behavior such as test configuration, test deployment and test schedule. The Interpreter defines the rules to select actions or parameters on certain events and conditions. In this way, given a set of knowledge at a certain time, a test agent dynamically adjusts its behavior according to its pre-defined rules and strategies. Case studies and experiments are exercised to apply the generic agent design to specific testing tasks such as performance testing and coverage-based testing.
@InProceedings{AST11p22,
author = {Xiaoying Bai and Bin Chen and Yunzhan Gong},
title = {Design of Intelligent Agents for Collaborative Testing of Service-Based Systems},
booktitle = {Proc.\ AST},
publisher = {ACM},
pages = {22--28},
doi = {},
year = {2011},
}
A Comparative Evaluation of State-of-the-Art Web Service Composition Testing Approaches
Hazlifah Mohd Rusli and
Sayed Gholam Hassan Tabatabaei
(Universiti Teknologi MARA, Malaysia; Universiti Teknologi Malaysia, Malaysia)
More and more Web based systems are being developed by composing other single or even composite services. This is due to the fact that not all available services are able to satisfy the needs of a user. The process of composing Web services involves discovering the appropriate services, selecting the best services, combining those services together, and finally executing them. Although much research efforts have been dedicated to the discovery, selection, and composition of services, the process of testing the Web service composition has not been given the same attention. This paper discusses the importance of Web services composition testing, provides a classification of the most prominent approaches in that area, presents several criteria for comparison of those approaches, and conducts a comparative evaluation of the approaches. The results of the paper give an essential perspective to do research work on Web services composition testing.
@InProceedings{AST11p29,
author = {Hazlifah Mohd Rusli and Sayed Gholam Hassan Tabatabaei},
title = {A Comparative Evaluation of State-of-the-Art Web Service Composition Testing Approaches},
booktitle = {Proc.\ AST},
publisher = {ACM},
pages = {29--35},
doi = {},
year = {2011},
}
Test Strategies I
A Comfortable TestPlayer for Analyzing Statistical Usage Testing Strategies
Winfried Dulz
(University of Erlangen-Nuremberg, Germany)
We will first give a brief introduction to a versatile modeling environment that allows the early validation of system specifications by a test-driven agile simulation approach. The main focus of our paper is on providing techniques for automated test case generation relying on statistical usage models. Based on the open source software R for statistical computing and graphics the easy to handle test case generation and analyzing Testplayer tool was developped. Starting from customer-specific graphical Markov chain usage models test cases are automatically generated and visualized by highlighting selected nodes and arcs in the model. In addition, various metrics and corresponding diagrams offer analytical techniques to assess the quality of the derived test suite.
@InProceedings{AST11p36,
author = {Winfried Dulz},
title = {A Comfortable TestPlayer for Analyzing Statistical Usage Testing Strategies},
booktitle = {Proc.\ AST},
publisher = {ACM},
pages = {36--42},
doi = {},
year = {2011},
}
Automatically Testing Interactive Multimodal Systems Using Task Trees and Fusion Models
Laya Madani and
Ioannis Parissis
(University Al Baath, Syria and Laboratoire d’Informatique de Grenoble, France; University of Grenoble and Grenoble INP - LCIS, France)
Multimodal systems support communication with the user
through different modalities such as voice and gesture. In
such systems, modalities may be used sequentially or concurrently,
and independently or combined synergistically. Their
use is characterized by four properties, called CARE (Complementarity,
Assignment, Redundancy and Equivalence).
This paper presents a method for generating automatically
test data for multimodal systems, based on two models: a
model describing the multimodal aspects and a task tree
model, both supporting operational profile specification.
@InProceedings{AST11p43,
author = {Laya Madani and Ioannis Parissis},
title = {Automatically Testing Interactive Multimodal Systems Using Task Trees and Fusion Models},
booktitle = {Proc.\ AST},
publisher = {ACM},
pages = {43--49},
doi = {},
year = {2011},
}
Using Conditional Mutation to Increase the Efficiency of Mutation Analysis
René Just, Gregory M. Kapfhammer, and
Franz Schweiggert
(Ulm University, Germany; Allegheny College, USA)
Assessing testing strategies and test sets is a crucial part of software testing. Mutation analysis is, among other approaches, a suitable technique for this purpose. However, compared with other methods it is rather time-consuming and applying mutation analysis to large software systems is still problematic. This paper presents a versatile approach, called conditional mutation, which increases the efficiency of mutation analysis. This new method significantly reduces the time overhead for generating and executing the mutants. Results are reported for eight investigated programs up to
373,000 lines of code and 406,000 generated mutants. Furthermore, conditional mutation has been integrated into the Java 6 Standard Edition compiler. Thus, it is widely applicable and not limited to a certain testing tool or framework.
@InProceedings{AST11p50,
author = {René Just and Gregory M. Kapfhammer and Franz Schweiggert},
title = {Using Conditional Mutation to Increase the Efficiency of Mutation Analysis},
booktitle = {Proc.\ AST},
publisher = {ACM},
pages = {50--56},
doi = {},
year = {2011},
}
Test Strategies II
Better Predicate Testing
Gary Kaminski and
Jeff Offutt
(George Mason University, USA)
Mutation testing is widely recognized as being extremely powerful, but is considered difficult to automate enough for practical use. This paper theoretically addresses two possible reasons for this: the generation of redundant mutants and the lack of integration of mutation analysis with other test criteria. By addressing these two issues, this paper brings an important mutation operator, relational-operator-replacement (ROR), closer to practical use. First, we develop fault hierarchies for the six relational operators, each of which generates seven mutants per clause. These hierarchies show that, for any given clause, only three mutants are necessary. This theoretical result can be integrated easily into mutation analysis tools, thereby eliminating generation of 57% of the ROR mutants. Second, we show how to bring the power of the ROR operator to the widely used Multiple Condition-Decision Coverage (MCDC) test criterion. This theoretical result includes an algorithm to transform any MCDC-adequate test set into a test set that also satisfies RORG, a new version of ROR appropriate for the MCDC context. The transformation does not use traditional mutation analysis, so can easily be integrated into existing MCDC tools and processes.
@InProceedings{AST11p57,
author = {Gary Kaminski and Jeff Offutt},
title = {Better Predicate Testing},
booktitle = {Proc.\ AST},
publisher = {ACM},
pages = {57--63},
doi = {},
year = {2011},
}
Hazard-based Selection of Test Cases
Mario Gleirscher
(TU München, Germany)
It is a challenge to engineer and select admissible test cases. This work approaches selection based on hazardous states through integrated modelling of the environment and the system, and by integration of requirements engineering activities into the tasks of system test engineering. The elaboration of test models and test case specifications focuses automation in system verification in order to detect hazard-oriented failures as well as in requirements validation in order to disclose potential defects in specification artefacts.
@InProceedings{AST11p64,
author = {Mario Gleirscher},
title = {Hazard-based Selection of Test Cases},
booktitle = {Proc.\ AST},
publisher = {ACM},
pages = {64--70},
doi = {},
year = {2011},
}
Applications of Testing I
Automated Testing of Industrial Control Devices: The Delphi Database
Nate Kube and
Daniel Hoffman
(Wurldtech Security Technologies, Canada; University of Victoria, Canada)
Delphi is a database designed to centralize and distribute current industrial automation vulnerability information. Over the past two years, with the aid of many of the world’s largest equipment vendors and operators, we have populated this database through extensive testing of industrial control devices. Delphi stores over 500 vulnerabilities on 31 popular distributed control system and safety instrumented system controllers. There are two primary reasons why Delphi data is useful: to distribute vulnerability data and to provide mitigation strategies. With detailed knowledge of which vulnerabilities are present, vendors can produce more robust devices and operators can construct business cases and calculate return-on-investment when considering investments in security measures. Furthermore, known vulnerabilities can be mitigated without applying device patches and shutting down plant operations.
@InProceedings{AST11p71,
author = {Nate Kube and Daniel Hoffman},
title = {Automated Testing of Industrial Control Devices: The Delphi Database},
booktitle = {Proc.\ AST},
publisher = {ACM},
pages = {71--76},
doi = {},
year = {2011},
}
Automating GUI Testing for Android Applications
Cuixiong Hu and
Iulian Neamtiu
(UC Riverside, USA)
Users increasingly rely on mobile applications for computational
needs. Google Android is a popular mobile platform, hence the
reliability of Android applications is becoming increasingly
important. Many Android correctness issues, however, fall outside the
scope of traditional verification techniques, as they are due to the
novelty of the platform and its
GUI-oriented application construction paradigm. In this paper we
present an approach for automating the testing process for Android
applications, with a focus on GUI bugs. We first conduct a bug mining
study to understand the nature and frequency of bugs affecting Android
applications; our study finds that GUI bugs are quite numerous. Next,
we present techniques for detecting GUI bugs by automatic generation
of test cases, feeding the application random events, instrumenting
the VM, producing log/trace files and analyzing them post-run. We show
how these techniques helped to re-discover existing bugs and find new
bugs, and how they could be used to prevent certain bug categories.
We believe our study and techniques have the potential to help
developers increase the quality of Android applications.
@InProceedings{AST11p77,
author = {Cuixiong Hu and Iulian Neamtiu},
title = {Automating GUI Testing for Android Applications},
booktitle = {Proc.\ AST},
publisher = {ACM},
pages = {77--83},
doi = {},
year = {2011},
}
Applications of Testing II
Selection and Execution of User Level Test Cases for Energy Cost Evaluation of Smartphones
Rajesh Palit and
Ajit Singh
(University of Waterloo, Canada)
Smartphones are emerging as a preferred communication device of users. In this paper, we provide a methodology to select user level test cases for performing energy cost evaluation of smartphone applications. We define the concept of a user level test case for smartphones and show that, due to configuration settings, there exist millions of such test cases. Next, we discuss a test selection technique to reduce the number of test cases. We apply the technique to five different smartphones and evaluate their energy costs for running common network related applications. We have developed a test bench to execute those test cases for real applications on smartphones and measure their actual energy costs. This work provides a framework for researchers and developers to conduct experiments for measuring the energy cost of applications on smartphones.
@InProceedings{AST11p84,
author = {Rajesh Palit and Ajit Singh},
title = {Selection and Execution of User Level Test Cases for Energy Cost Evaluation of Smartphones},
booktitle = {Proc.\ AST},
publisher = {ACM},
pages = {84--90},
doi = {},
year = {2011},
}
Testing an Optimising Compiler by Generating Random Lambda Terms
Michał H. Pałka,
Koen Claessen,
Alejandro Russo, and John Hughes
(Chalmers University of Technology, Sweden; Quviq AB, Sweden)
@InProceedings{AST11p91,
author = {Michał H. Pałka and Koen Claessen and Alejandro Russo and John Hughes},
title = {Testing an Optimising Compiler by Generating Random Lambda Terms},
booktitle = {Proc.\ AST},
publisher = {ACM},
pages = {91--90},
doi = {},
year = {2011},
}
Model-Driven Design and Validation of Embedded Software
Giuseppe Di Guglielmo, Masahiro Fujita, Luigi Di Guglielmo, and
Andreas Foltinek
(The University of Tokyo, Japan; University of Verona, Italy; STM Products srl, Italy; IMACS GmbH, Germany)
@InProceedings{AST11p98,
author = {Giuseppe Di Guglielmo and Masahiro Fujita and Luigi Di Guglielmo and Andreas Foltinek},
title = {Model-Driven Design and Validation of Embedded Software},
booktitle = {Proc.\ AST},
publisher = {ACM},
pages = {98--97},
doi = {},
year = {2011},
}
Poster Session 1
Test Data to Reduce the Complexity of Unit Test Automation
Guy Collins Ndem, Abbas Tahir, and
Helmut Goetz
(Siemens AG, Germany)
In order to automatically test large and complex software systems, a well defined test process and a huge amount of test data are needed. When it comes to test automation, the quality of test data always plays a significant role by reducing the cost and the time required for the test activities. In this work, we follow the principle of "Quality of the test cases comes before Quantity" to show how the right selection of test data helps to address the goal of qualitative testing while optimizing the test effort needed
@InProceedings{AST11p105,
author = {Guy Collins Ndem and Abbas Tahir and Helmut Goetz},
title = {Test Data to Reduce the Complexity of Unit Test Automation},
booktitle = {Proc.\ AST},
publisher = {ACM},
pages = {105--106},
doi = {},
year = {2011},
}
A Framework for Automatic Functional Testing Based on Formal Specifications
Shaoying Liu and
Shin Nakajima
(Hosei University, Japan; NII, Japan)
This paper describes a framework for automatic functional testing based on formal specifications. The framework is intended to expose all of the issues that must be addressed in automatic testing, to provide solutions for some of the major challenges, and to serve as a foundation for developing tool support in the future.
@InProceedings{AST11p107,
author = {Shaoying Liu and Shin Nakajima},
title = {A Framework for Automatic Functional Testing Based on Formal Specifications},
booktitle = {Proc.\ AST},
publisher = {ACM},
pages = {107--108},
doi = {},
year = {2011},
}
Poster Session 2
Towards Automated Testing of Web Service Choreographies
Felipe M. Besson, Pedro M. B. Leal, Fabio Kon, Alfredo Goldman, and Dejan Milojicic
(University of São Paulo, Brazil; Hewlett Packard Laboratories, USA)
Web service choreographies have been proposed as a decentralized scalable way of composing services in a SOA environment. In spite of all the benefits of choreographies, the decentralized flow of information, the parallelism, and multiple party communication restrict
the automated testing of choreographies at design and runtime. The goal of our research is to adapt the automated testing techniques used by the Agile Software Development community to the SOA context. To achieve that, we seek to develop software tools and a methodology to enable test-driven development of Choreographies. In this paper, we present our first step in that direction, a software prototype composed of ad hoc automated test case scripts for testing a web service choreography.
@InProceedings{AST11p109,
author = {Felipe M. Besson and Pedro M. B. Leal and Fabio Kon and Alfredo Goldman and Dejan Milojicic},
title = {Towards Automated Testing of Web Service Choreographies},
booktitle = {Proc.\ AST},
publisher = {ACM},
pages = {109--110},
doi = {},
year = {2011},
}
monadWS: A Monad-Based Testing Tool for Web Services
Yingzhou Zhang and
Changhai Nie
(Nanjing University of Posts and Telecommunications, China; Nanjing University, China)
Testing Web Services (WS) is very important because it could guarantee a high degree of service quality and reliability. Automatic WS testing tools enable testers to complete testing in a shorter time, and they make it easy to repeat tests after each modification to a service. In this paper, we present monadWS, a prototype tool of monad-based automated testing for WS, which enables concrete description, autogeneration and execution for WS test cases with the design of service testing monads. The results show that monadWS is feasible and effective for testing WS.
@InProceedings{AST11p111,
author = {Yingzhou Zhang and Changhai Nie},
title = {monadWS: A Monad-Based Testing Tool for Web Services},
booktitle = {Proc.\ AST},
publisher = {ACM},
pages = {111--112},
doi = {},
year = {2011},
}
Towards Automated Oracles for GUI Input Validation
Gabriel L. Zenarosa and
Regis J. Leonard
(University of Pittsburgh, USA)
Testing input validation in web applications from specifications is a challenging and laborious process. GUI testing tools---with their record-and-playback and data-driven capabilities---ease the pains of testing through automation. Out-of-the-box, however, these tools have some scaling limitations as setup costs are incurred for every distinct web application to test. In environments where a line of many web applications are regularly created for various customers and purposes, scaling the test automation to span the entire product line is extremely valuable. In this paper, we report on our experience in generalizing the automatic specification-based testing of input validation in a line of web applications. Our approach is based on a nonstandard use of a GUI testing tool enabled by adjustments to coding standards and the requirements specification writing process.
@InProceedings{AST11p113,
author = {Gabriel L. Zenarosa and Regis J. Leonard},
title = {Towards Automated Oracles for GUI Input Validation},
booktitle = {Proc.\ AST},
publisher = {ACM},
pages = {113--114},
doi = {},
year = {2011},
}
proc time: 0.05