ICSE 2013 Workshops
2013 35th International Conference on Software Engineering (ICSE)
Powered by
Conference Publishing Consulting

2013 8th International Workshop on Automation of Software Test (AST), May 18-19, 2013, San Francisco, CA, USA

AST 2013 – Proceedings

Contents - Abstracts - Authors

8th International Workshop on Automation of Software Test (AST)

Preface

Title Page


Foreword
The Workshops on the Automation of Software Test (AST) focus on bridging the gap between the theory and practice of software test automation. It aims at providing researchers and practitioners with a forum for exchanging ideas and experiences, developing an understanding of the fundamental challenges, articulating a vision for the future, and finding promising solutions to pressing problems. AST 2013 is the 8th edition of the workshops since the first one in 2006. The authors are invited to submit their original work on topics including, but not limited to, the methodology, technology development and transfer, software tools and environments, and experience reports related to the automation of software testing. In addition to regular research papers, the workshop will also have a case study track to feature papers reporting the state of practice in the automation of software testing. In an effort to enable participants to propose and prioritize challenges and to interactively work out potential solutions, the workshop also has a charette discussion session, which focuses on a special important topic. The special topic for AST 2013 is on Testing of Software as a Service.

Testing-as-a-Service I

Integrated TaaS Platform for Mobile Development: Architecture Solutions
Oleksii Starov and Sergiy Vilkomir
(East Carolina University, USA)
This paper examines the Testing-as-a-Service (TaaS) solutions in mobile development and proposes a universal TaaS platform: Cloud Testing of Mobile Systems (CTOMS). CTOMS is an integrated solution with a core infrastructure that enables the scaling of additional functionalities. The CTOMS’s benefits are explained, the architecture of the system is described in detail, and technical solutions are listed based on the feasibility study that resulted in creation of the first version of CTOMS for Android development.

Automated Scalability Testing of Software as a Service
Paulo Moura and Fabio Kon
(USP, Brazil)
Scalability has been studied in several areas of Computer Science and scalability testing and evaluation of con- temporary software systems is an active topic. However, most of the times, these activities are still performed in a predominantly ad hoc fashion. There are a few tools to automate this process, but they present several restrictions about what systems can be tested and how to evaluate scalability. In this paper, we introduce a flexible and extensible framework for automated scalability testing of software offered as a service and propose to evaluate the scalability using hypothesis tests. Additionally, we argue that, instead of stating if a system is scalable or not, we should find out how it could scale better.

Vee@Cloud: The Virtual Test Lab on the Cloud
Xiaoying Bai, Muyang Li, Xiaofei Huang, Wei-Tek Tsai, and Jerry Gao
(Tsinghua University, China; Arizona State University, USA; San Jose State University, USA)
Large-scale system testing is challenging. It usually requires large number of test cases, substantial resources, and geographical distributed usage scenarios. It is expensive to build the test environment and to achieve certain level of test confidence. To address the challenges, test systems need to be scalable in a cost-effective manner. TaaS (Testing-as-a-Service) promotes a Cloud-based testing architecture to provide online testing services following a pay-per-use business model. The paper introduces the research and implementation of a TaaS system called Vee@Cloud. It serves as a scalable virtual test lab built upon Cloud infrastructure services. The resource manager allocates Virtual Machine instances and deploy test tasks, from a pool of available resources across different Clouds. The workload generator simulates various workload patterns, especially for system with new architecture styles like Web 2.0 and big data processing. Vee@Cloud promotes continuous monitoring and evaluating of online services. The monitor collects real-time performance data and analyzes the data against SLA (Service Level Agreement). A proof-of-concept prototype system is built and some early experiments are exercised using public Cloud services.

Testing-as-a-Service II

Test Algebra for Combinatorial Testing
Wei-Tek Tsai, Charles J. Colbourn, Jie Luo, Guanqiu Qi, Qingyang Li, and Xiaoying Bai
(Arizona State University, USA; Beihang University, China; Tsinghua University, China)
This paper proposes a new algebraic system, Test Algebra (TA), for identifying faults in combinatorial testing for SaaS (Software-as-a-Service) applications. SaaS as a part of cloud computing is a new software delivery model, and mission-critical applications are composed, deployed, and executed in cloud platforms. Testing SaaS applications is a challenging task because new applications need to be tested when they are composed before they can be deployed for execution. Combinatorial testing algorithms can be used to identify faulty configurations and interactions from 2-way all the way to k-way where k is the number of components in the application. The TA defines rules to identify faulty configurations and interactions. Using the rules defined in the TA, a collection of configurations can be tested concurrently in different servers and in any order and the results obtained will be still same due to the algebraic constraints.

ParTes: A Test Generation Strategy for Choreography Participants
Francesco De Angelis, Daniele Fanì, and Andrea Polini
(University of Camerino, Italy; ISTI - CNR, Italy)
Inter-organizational business processes permit to specify how different organizations can integrate to carry on business activities together. In this context choreography specifica- tions provide a particularly useful view permitting to define how different organizations should interact and exchange messages in order to fruitfully cooperate. Tools and mechanisms permitting to check that a stakeholder, and its provided e-services, are able to correctly cooperate according to the global specification become an important and useful asset. This is particularly true when open specifications are considered and services dynamically integrate with each other at run-time.
This paper proposes a novel derivation strategy for test case skeletons, which can be successively refined and concretized to check the behaviour of parties willing to play a role within a choreography enactment. The very basic idea is to derive test cases from the possible interaction traces included in the choreography specification handled as a workflow graph. The selection of traces to use for test derivation purpose is driven by a specifically conceived technique to work flow refactoring which permits to reduce the number of interleavings to explore, in particular when parallel statements are considered.

Functional SOA Testing Based on Constraints
Seema Jehan, Ingo Pill, and Franz Wotawa
(TU Graz, Austria)
In the fierce competition on today's software market, Service-Oriented Architectures (SOAs) are an established design paradigm. Essential concepts like modularization, reuse, and the corresponding IP core business are inherently supported in the development and operation of SOAs that offer flexibility in many aspects and thus optimal conditions also for heterogeneous system developments. The intrinsics of large and complex SOA enterprises, however, require us to adopt and evolve our verification technology, in order to achieve expected software quality levels. In this paper, we contribute to this challenge by proposing a constraint based testing approach for SOAs. In our work, we augment a SOA's BPEL business model with pre- and postcondition contracts defining essential component traits, and derive a suite of feasible test cases to be executed after assessing its quality via corresponding coverage criteria. We illustrate our approach's viability via a running example as well as experimental results, and discuss current and envisioned automation levels in the context of a test and diagnosis workflow.

Testing Distributed, Concurrent, and Parallel Systems

Automatic Generation of Parallel Unit Tests
Jochen Schimmel, Korbinian Molitorisz, Ali Jannesari, and Walter F. Tichy
(KIT, Germany; German Research School for Simulation Sciences, Germany; RWTH Aachen University, Germany)
Multithreaded software is subject to data races. Currently available data race detectors report such errors to the developer, but consume large amounts of time and memory; many approaches are not applicable for large software projects. Unit tests containing fractions of the program lead to better results.We propose AutoRT, an approach to automatically generate parallel unit tests as target for data race detectors from existing programs. AutoRT uses the Single Static Multiple Dynamic (SSMD) analysis pattern to reduce complexity and can therefore be used efficiently even in large software projects. We evaluate AutoRT using Microsoft CHESS and show that with SSMD all 110 data races contained in our sample programs can be located.

Profile Directed Systematic Testing of Concurrent Programs
Yan Hu, Jun Yan, Jian Zhang, and He Jiang
(Dalian University of Technology, China; ISCAS, China)
Runtime data is a rich source of feedback information which can be used to improve program analysis. In this paper, we proposed a Profile directed Event driven Dynamic AnaLysis (PEDAL) to effectively detect concurrency bugs. PEDAL identifies important schedule points with the help of profiling data, and generates a reduced set of schedule points where preemptions could happen. The reduced preemption set is then used to direct the search for erroneous schedules. PEDAL is evaluated on a set of multithreaded benchmark programs, including MySQL, the industrial level database server application. Experimental results show that PEDAL is both efficient and scalable, as compared with several existing analysis techniques.

Which Compiler Optimization Options Should I Use for Detecting Data Races in Multithreaded Programs?
Changjiang Jia and W. K. Chan
(City University of Hong Kong, China)
Different compiler optimization options may produce different versions of object code. To the best of our knowledge, existing studies on concurrency bug detection in the public literature have not reported the effects of different compiler optimization options on detection effectiveness. This paper reports a preliminary but the first study in the exploratory nature to investigate this aspect. The study examines the happened-before based predictive data race detection scenarios on four benchmarks from the PARSEC 3.0 suite compiled under six different GNU GCC optimization options. We observe from the data set that the same race detection technique may produce different sets of races or different detection probabilities under different optimization scenarios. Based on the observations, we formulate two hypotheses for future investigations.

Security Testing

Security Testing of the Communication among Android Applications
Andrea Avancini and Mariano Ceccato
(Fondazione Bruno Kessler, Italy)
An important reason behind the popularity of smartphones and tablets is the huge amount of available applications to download, to expand functionalities of the devices with brand new features. In fact, official stores provide a plethora of applications developed by third parties, for entertainment and business, most of which for free. However, confidential data (e.g., phone contacts, global GPS position, banking data and emails) could be disclosed by vulnerable applications. Sensitive applications should carefully validate exchanged data to avoid security problems. In this paper, we propose a novel testing approach to test communication among applications on mobile devices. We present a test case generation strategy and a testing adequacy criterion for Android applications. Our approach has been assessed on three widely used Android applications.

Access Control Enforcement Testing
Donia El Kateb, Yehia ElRakaiby, Tejeddine Mouelhi, and Yves Le Traon
(University of Luxembourg, Luxembourg)
A policy-based access control architecture comprises Policy Enforcement Points (PEPs), which are modules that intercept subjects access requests and enforce the access decision reached by a Policy Decision Point (PDP), the module implementing the access decision logic. In applications, PEPs are generally implemented manually, which can introduce errors in policy enforcement and lead to security vulnerabilities. In this paper, we propose an approach to systematically test and validate the correct enforcement of access control policies in a given target application. More specifically, we rely on a two folded approach where a static analysis of the target application is first made to identify the sensitive accesses that could be regulated by the policy. The dynamic analysis of the application is then conducted using mutation to verify for every sensitive access whether the policy is correctly enforced. The dynamic analysis of the application also gives the exact location of the PEP to enable fixing enforcement errors detected by the analysis. The approach has been validated using a case study implementing an access control policy.

XSS Pattern for Attack Modeling in Testing
Josip Bozic and Franz Wotawa
(TU Graz, Austria)
Security issues of web applications are still a current topic of interest especially when considering the consequences of unintended behaviour. Such services might handle sensitive data about several thousands or millions of users. Hence, exploiting services or other undesired effects that cause harm on users has to be avoided. Therefore, for software developers of such applications one of the major tasks in providing security is to embed testing methodologies into the software development cycle, thus minimizing the subsequent damage resulting in debugging and time intensive upgrading. Model-based testing evolved as one of the methodologies which offer several theoretical and practical approaches in testing the system under test (SUT) that combine several input generation strategies like mutation testing, using of concrete and symbolic execution etc. by putting the emphasis on specification of the model of an application. In this work we propose an approach that makes use of an attack pattern model in form of a UML state machine for test case generation and execution. The paper also discusses the current implementation of our attack pattern testing tool using a XSS attack pattern and demonstrates the execution in a case study.

Test-Case Generation

Automated Generation of State Abstraction Functions using Data Invariant Inference
Paolo Tonella, Cu Duy Nguyen, Alessandro Marchetto, Kiran Lakhotia, and Mark HarmanORCID logo
(Fondazione Bruno Kessler, Italy; University College London, UK)
Model based testing relies on the availability of models that can be defined manually or by means of model inference techniques. To generate models that include meaningful state abstractions, model inference requires a set of abstraction functions as input. However, their specification is difficult and involves substantial manual effort. In this paper, we investigate a technique to automatically infer both the abstraction functions necessary to perform state abstraction and the finite state models based on such abstractions. The proposed approach uses a combi- nation of clustering, invariant inference and genetic algorithms to optimize the abstraction functions along three quality attributes that characterize the resulting models: size, determinism and infeasibility of the admitted behaviors. Preliminary results on a small e-commerce application are extremely encouraging because the automatically produced models include the set of manually defined gold standard models.

Towards Automatic Scenario Generation from Coverage Information
Melanie Diepenbeck, Mathias Soeken, Daniel Große, and Rolf Drechsler
(University of Bremen, Germany; DFKI at Bremen, Germany; solvertec, Germany)
Nowadays, the design of software systems is pushed towards agile development practices. One of its most fundamental approaches is Test Driven Development (TDD). This procedure is based on test cases which are incrementally written prior to the implementation. Recently, Behavior Driven Development (BDD) has been introduced as an extension of TDD, in which natural language scenarios are the starting point for the test cases. This description offers a ubiquitous communication mean for both the software developers and stakeholders. Following the BDD methodology thoroughly, one would expect 100% code coverage, since code is only written to make the test cases pass. However, as we show in an empirical study this expectation is not valid in practice. It becomes even worse in the process of development, i.e. the coverage decreases over time. To close the coverage gap, we sketch an algorithm that generates BDD-style scenarios based on uncovered code.

Test-Duo: A Framework for Generating and Executing Automated Acceptance Tests from Use Cases
Chin-Yun Hsieh, Chen-Hsin Tsai, and Yu Chin Cheng
(National Taipei University of Technology, Taiwan)
A new approach to generate automated acceptance test cases from use cases in order to alleviate the tedious work of creating and performing acceptance tests is presented. In the proposed approach, annotations are adopted as markers to define system behaviors and make the use case explicit. With the annotated use cases, ROBOT framework compatible test cases can be generated and applied to test the system. A supporting tool is also constructed to assist overall approach proposed in this paper.

Reuse, Refactoring, and Regression Test

A Refactoring-Based Approach for Test Case Selection and Prioritization
Everton L. G. Alves, Patricia D. L. Machado, Tiago Massoni, and Samuel T. C. Santos
(UFCG, Brazil)
Refactoring edits, commonly applied during software development, may introduce faults in a previously-stable code. Therefore, regression testing is usually applied to check whether the code maintains its previous behavior. In order to avoid rerunning the whole regression suite, test case prioritization techniques have been developed to order test cases for earlier achievement of a given goal, for instance, improving the rate of fault detection during regression testing execution. However, as current techniques are usually general purpose, they may not be effective for early detection of refactoring faults. In this paper, we propose a refactoring-based approach for selecting and prioritizing regression test cases, which specializes selection/prioritization tasks according to the type of edit made. The approach has been evaluated through a case study that compares it to well-known prioritization techniques by using a real open-source Java system. This case study indicates that the approach can be more suitable for early detection of refactoring faults when comparing to the other prioritization techniques.

Utilizing Software Reuse Experience for Automated Test Recommendation
Werner Janjic and Colin Atkinson
(University of Mannheim, Germany)
The development of defect tests is still a very labour intensive process that demands a high-level of domain knowledge, concentration and problem awareness from software engineers. Any technology that can reduce the manual effort involved in this process therefore has the potential to significantly reduce software development costs and time consumption. An idea for achieving this is to reuse the knowledge bound up in already existing test cases, either directly or indirectly, to assist in the development of tests for new software components and systems. Although general software reuse has received a lot of attention in the past - both in academia and industry - previous research has focussed on the reuse and recommendation of existing software artifacts in the creation of new product code rather than on the recommendation of tests. In this paper we focus on the latter, and present a novel automated test recommendation approach that leverages lessons learned from traditional software reuse to proactively make test case suggestions while an engineer is developing tests. In contrast, most existing testing-assistance tools provide ex post assistance to test developers in the form of coverage assessments and test quality evaluations. Our goal is to create an automated, non-intrusive recommendation system for efficient software test development. In this paper we set out the basic strategy by which this can be achieved and present a prototypical implementation of our test recommender system for Eclipse.

Did We Test Our Changes? Assessing Alignment between Tests and Development in Practice
Sebastian Eder, Benedikt Hauptmann, Maximilian Junker, Elmar Juergens, Rudolf Vaas, and Karl-Heinz Prommer
(TU Munich, Germany; CQSE, Germany; Munich Re, Germany)
Testing and development are increasingly performed by different organizations, often in different countries and time zones. Since their distance complicates communication, close alignment between development and testing becomes increasingly challenging. Unfortunately, poor alignment between the two threatens to decrease test effectiveness or increases costs. In this paper, we propose a conceptually simple approach to assess test alignment by uncovering methods that were changed but never executed during testing. The paper's contribution is a large industrial case study that analyzes development changes, test service activity and field faults of an industrial business information system over 14 months. It demonstrates that the approach is suitable to produce meaningful data and supports test alignment in practice.

Application Software Testing

Automatic Test Generation for Mutation Testing on Database Applications
Kai Pan, Xintao Wu, and Tao Xie
(University of North Carolina at Charlotte, USA; North Carolina State University, USA)
To assure high quality of database applications, testing database applications remains the most popularly used approach. In testing database applications, tests consist of both program inputs and database states. Assessing the adequacy of tests allows targeted generation of new tests for improving their adequacy (e.g., fault-detection capabilities). Comparing to code coverage criteria, mutation testing has been a stronger criterion for assessing the adequacy of tests. Mutation testing would produce a set of mutants (each being the software under test systematically seeded with a small fault) and then measure how high percentage of these mutants are killed (i.e., detected) by the tests under assessment. However, existing test-generation approaches for database applications do not provide sufficient support for killing mutants in database applications (in either program code or its embedded or resulted SQL queries). To address such issues, in this paper, we propose an approach called MutaGen that conducts test generation for mutation testing on database applications. In our approach, we first apply an existing approach that correlates various constraints within a database application through constructing synthesized database interactions and transforming the constraints from SQL queries into normal program code. Based on the transformed code, we generate program-code mutants and SQL-query mutants, and then derive and incorporate query-mutant-killing constraints into the transformed code. Then, we generate tests to satisfy query-mutant-killing constraints. Evaluation results show that MutaGen can effectively kill mutants in database applications, and MutaGen outperforms existing test-generation approaches for database applications in terms of strong mutant killing.

An Industry Proof-of-Concept Demonstration of Automated Combinatorial Test
Redge Bartholomew
(Rockwell Collins, USA)
Abstract—Studies have found that the largest single cost and schedule component of safety-critical, embedded system development is software rework: locating and fixing software defects found during test. In many such systems these defects are the result of interactions among no more than 6 variables, suggesting that 6-way combinatorial testing would be sufficient to trigger and detect them. The National Institute of Standards and Technology developed an approach to automatically generating, executing, and analyzing such tests. This paper describes an industry proof-of-concept demonstration of automated unit and integration testing using this approach. The goal was to see if it might cost-effectively reduce rework by reducing the number of software defects escaping into system test – if it was adequately accurate, scalable, mature, easy to learn, and easy to use and still was able to achieve the required level of structural coverage. Results were positive – e.g., 2775 test input vectors were generated in 6 seconds, expected outputs were generated in 60 minutes, and executing and analyzing them took 8 minutes. Tests detected all seeded defects and in the proof-of-concept demonstration achieved nearly 100% structural coverage.

Extension of Selenium RC Tool to Perform Automated Testing with Databases in Web Applications
Andreza M. F. V. de Castro, Gisele A. Macedo, Eliane F. Collins, and Arilo C. Dias-Neto
(Nokia Institute of Technology, Brazil; UFAM, Brazil)
This paper presents one extension of the Selenium RC tool to perform tests in web applications that require checking data in databases. This verification is performed using new functions implemented into the Selenium frameworks core. These functions allow opening and closing database connection and comparing test data with data stored in the database used by the web application. This solution aims to contribute to the system quality by reducing the effort during the testing process, since the verification of UI and database elements will be performed at the same time during execution of the test scripts. A case study is described to analyze the impact of the proposed tool in terms of effort and rate of automation in the development of a new web application project. The results suggest a significant reduction (92% and 88%) in the effort to execute automated tests in database when compared to, respectively, manual and semi-automated execution.

Testing and Debugging

Quantifying the Complexity of Dataflow Testing
Giovanni Denaro, Mauro PezzèORCID logo, and Mattia Vivanti
(University of Milano-Bicocca, Italy; University of Lugano, Switzerland)
It is common belief that dataflow testing criteria are harder to satisfy than statement and branch coverage. As motivations, several researchers indicate the difficulty of finding test suites that exercise many dataflow relations and the increased impact of infeasible program paths on the maximum coverage rates that can be indeed obtained. Yet, although some examples are given in research papers, we lack data on the validity of these hypotheses. This paper presents an experiment with a large sample of object oriented classes and provides solid empirical evidence that dataflow coverage rates are steadily lower than statement and branch coverage rates, and that the uncovered dataflow elements do not generally depend on the feasibility of single statements.

Computation and Visualization of Cause-Effect Paths
Alpana Dubey and P. V. R. Murthy
(ABB, India; Siemens, India)
Static analyzers detect possible run-time errors at compile-time and often employ data-ow analysis techniques to infer properties of programs. Usually, dataow analysis tools report possible errors with line numbers in source code and leave the task of locating root causes of errors. This paper proposes a technique to aid developers in locating the root causes of statically identied run-time errors with the help of cause-effect paths. A cause effect path terminates at an erroneous statement and originates at the statement which is responsible for the error. We propose modications to the classic data-ow analysis algorithm to compute cause-effect paths. We discuss different visualization modes in which cause-effect paths can be displayed. As a case study, we implemented a null pointer analyzer, with the additional capability of cause-effect path computation, using the Microsoft Phoenix framework. In addition, we propose a methodology to automatically generate an analyzer which computes cause-effect paths using a framework such as Microsoft Phoenix.

JIFFY: A Framework for Encompassing Aspects in Testing and Debugging Software
Mohammad Asif and Y. Raghu Reddy
(IIIT Hyderabad, India)
Aspect Oriented Programming (AOP) advocates the notion of aspects to encapsulate crosscutting concerns. A concern is a behavior in a computer program and is said to be crosscutting if the module(s) that address the behavior are scattered and tangled with other modules of the system. In this paper, we investigate the possibility of using AOP for software testing and non-invasive debugging. We have identified some crosscutting concerns like access control, logging, performance, tracing, etc., and developed a framework based on java bytecode instrumentation technique to inject these crosscutting concerns into the compiled code. The framework is available as a service, i.e., the service takes the required code as input and produces the changed code as output that contains the appropriate aspects. Thus, it is argued that the framework can be the basis for implementing the notion of Enabling testing as a Service.

proc time: 1.41