SBST 2014 – Proceedings

Message from the Chairs
Welcome to the 7th International Workshop on Search-Based Software Testing (SBST 2014), held in Hyderabad, India on June 2–3 2014, co-located with ICSE 2014. Search-Based Software Testing (SBST) is a form of Search-Based Software Engineering (SBSE) that optimizes testing through computational search, and the SBST workshop is a venue for recent results and high quality discussions regarding the current and future state of the field.

Short and Full Papers

Test Generation across Multiple Layers
Matthias Höschele, Juan Pablo Galeotti, and Andreas Zeller
(Saarland University, Germany)
Complex software systems frequently come in many layers, each realized in a different programming language. This is a challenge for test generation, as the semantics of each layer have to be determined and integrated. An automatic test generator for Java, for instance, is typically unable to deal with the internals of lower level code (such as C-code), which results in lower coverage and fewer test cases of interest. In this paper, we sketch a novel approach to help search-based test generators for Java to achieve better coverage of underlying native code layers. The key idea is to apply test generation to the native layer first, and then to use the inputs to the native test cases as targets for search-based testing of the higher Java layers. We demonstrate our approach on a case study combining KLEE and EVOSUITE.

Search-Based Security Testing of Web Applications
Julian Thomé, Alessandra Gorla, and Andreas Zeller
(Saarland University, Germany)
SQL injections are still the most exploited web application vulnerabilities. We present a technique to automatically detect such vulnerabilities through targeted test generation. Our approach uses search-based testing to systematically evolve inputs to maximize their potential to expose vulnerabilities. Starting from an entry URL, our BIOFUZZ prototype systematically crawls a web application and generates inputs whose effects on the SQL interaction are assessed at the interface between Web server and database. By evolving those inputs whose resulting SQL interactions show best potential, BIOFUZZ exposes vulnerabilities on real-world Web applications within minutes. As a black-box approach, BIOFUZZ requires neither analysis nor instrumentation of server code; however, it even outperforms state-of-the-art white-box vulnerability scanners.

Root Cause Analysis for HTML Presentation Failures using Search-Based Techniques
Sonal Mahajan, Bailan Li, and William G. J. Halfond
(University of Southern California, USA)
Presentation failures in web applications can negatively impact users' perception of the application's quality and its usability. Such failures are challenging to diagnose and correct since the user interfaces of modern web applications are defined by a complex interaction between HTML tags and their visual properties defined by CSS and HTML attributes. In this paper, we introduce a novel approach for automatically identifying the root cause of presentation failures in web applications that uses image processing and search based techniques. In an experiment conducted for assessing the accuracy of our approach, we found that it was able to identify the correct root cause with 100% accuracy.

Moving the Goalposts: Coverage Satisfaction Is Not Enough
Gregory Gay, Matt Staats, Michael W. Whalen, and Mats P. E. Heimdahl
(University of Minnesota, USA; University of Luxembourg, Luxembourg)
Structural coverage criteria have been proposed to measure the adequacy of testing efforts. Indeed, in some domains—e.g., critical systems areas—structural coverage criteria must be satisfied to achieve certification. The advent of powerful search-based test generation tools has given us the ability to generate test inputs to satisfy these structural coverage criteria. While tempting, recent empirical evidence indicates these tools should be used with caution, as merely achieving high structural coverage is not necessarily indicative of high fault detection ability. In this report, we review some of these findings, and offer recommendations on how the strengths of search-based test generation methods can alleviate these issues.

Code Hunt: Searching for Secret Code for Fun
Nikolai Tillmann, Judith Bishop, Nigel Horspool, Daniel Perelman, and Tao Xie
(Microsoft Research, USA; University of Victoria, Canada; University of Washington, USA; University of Illinois at Urbana-Champaign, USA)
Learning to code can be made more effective and sustainable if it is perceived as fun by the learner. Code Hunt uses puzzles that players have to explore by means of clues presented as test cases. Players iteratively modify their code to match the functional behavior of secret solutions. This way of learning to code is very different to learning from a specification. It is essentially re-engineering from test cases. Code Hunt is based on the test/clue generation of Pex, a white-box test generation tool that uses dynamic symbolic execution. Pex performs a guided search to determine feasible execution paths. Conceptually, solving a puzzle is the manual process of conducting search-based test generation: the “test data” to be generated by the player is the player’s code, and the “fitness values” that reflect the closeness of the player’s code to the secret code are the clues (i.e., Pex-generated test cases). This paper is the first one to describe Code Hunt and its extensions over its precursor Pex4Fun. Code Hunt represents a high-impact educational gaming platform that not only internally leverages fitness values to guide test/clue generation but also externally offers fun user experiences where search-based test generation is manually emulated. Because the amount of data is growing all the time, the entire system runs in the cloud on Windows Azure.

An Overview of Search Based Combinatorial Testing
Huayao Wu and Changhai Nie
(Nanjing University, China)
Combinatorial testing (CT) is a branch of software testing, which aims to detect the interaction triggered failures as much as possible. Search based combinatorial testing is to use the search techniques to solve the problem in combinatorial testing. It has been shown to be effective and promising. In this paper, we aim to provide an overview of search based combinatorial testing, especially focusing on test suite generation without constraint, and discuss the potential future directions in this field.

PhD Papers

Model Based Test Case Generation with Metaheuristics for Networks of Timed Automata
Joachim Hänsel
(HPI, Germany)
Model-Based Testing, the task of generating test inputs and oracles from a test model, has been successfully applied in the context of safety-critical real time systems. As these systems grow in complexity, test-models, designed to reflect the systems behaviour, will grow too. Currently testers face situations where test-models are too complex for present test generators.
In this paper, we outline a software tool for the evaluation of the scalability of a combination of approaches for model-based test generation. We chose Networks of Timed Automata (NTA) as the modeling formalism because real-time properties can be specified and the semantics are well-defined. However, the tool input is given as a restricted UML statechart which is internally transformed. We expect this to increase industrial acceptance. The tool will provide the selection, parametrization and generation of a metaheuristic algorithm. The aim is to support test model specific generation algorithms. A simulator for NTAs will enable the metaheuristic to search for test goals in the model. For better performance, it will have an advanced parallelisation. Furthermore, input models will be used for search space reduction for even faster test case generation. The proposed approach allows the inclusion of an oracle generator that is able to provide expected outputs; this enables conformance checking between test models and systems under test.
We plan to implement the outlined tool to enable test case generation even for models that are beyond the scope of currently available generators.

Position Papers

Search Based Techniques for Software Fault Prediction: Current Trends and Future Directions
Ruchika Malhotra
(Delhi Technological University, India)
The effective allocation of the resources is crucial and essential in the testing phase of the software development life cycle so that the weak areas in the software can be verified and validated efficiently. The prediction of fault prone classes in the early phases of software development can help software developers to focus the limited available resources on those portions of software, which are more prone to fault. Recently, the search based techniques have been successfully applied in the software engineering domain. In this study, we analyze the position of search based techniques for use in software fault prediction by collecting relevant studies from the literature which were conducted during the period January 1991 to October 2013. We further summarize current trends by assessing the performance capability of the search based techniques in the existing research and suggest future directions.

A Hybrid Test Optimization Framework using Memetic Algorithm with Cuckoo Flocking Based Search Approach
Jeya Mala Dharmalingam, Sabarinathan K., and Balamurugan S.
(Thiagarajar College of Engineering, India; Tata Consultancy Services, India)
The testing process of industrial strength applications usually takes more time to ensure that all the components are rigorously tested to have failure-free operation upon delivery. This research work proposed a hybrid optimization approach that combines the population based multi-objective optimization approach namely Memetic Algorithm with Cuckoo Search (MA-CK) to generate optimal number of test cases that achieves the specified test adequacy criteria based on mutation score and branch coverage. Further, GA, HGA and MA based heuristic algorithms are empirically evaluated and it has been shown that the proposed MA with cuckoo search based optimization algorithm provides an optimal solution.

SBST 2014 – Proceedings

7th International Workshop on Search-Based Software Testing (SBST 2014)

Frontmatter

Short and Full Papers

PhD Papers

Position Papers