Powered by
Conference Publishing Consulting

2013 International Workshop on Joining AcadeMiA and Industry Contributions to testing Automation (JAMAICA), July 15, 2013, Lugano, Switzerland

JAMAICA 2013 – Proceedings

Contents - Abstracts - Authors

2013 International Workshop on Joining AcadeMiA and Industry Contributions to testing Automation (JAMAICA)

Preface

Title Page


Foreword
It is our great pleasure to welcome you to the first edition of JAMAICA, the first international workshop on “Joining AcadeMiA and Industry Contributions to testing Automation”.

Testing Techniques

Early Detection of Faults Related to Database Schematic Changes
Deepak Garg and Amitava Datta
(University of Western Australia, Australia)
Modern software applications consisting of a multi-tiered application structure have often a database as an essential tier. The correct functioning of a multi-tiered application is dependent on its correct database connectivity. The modifications in the schema of the database may result in faults and may end up in the non-functionality of a software application. To test for the modifications in the database schema, testers normally execute all the test cases for the application. In this paper, we primarily focus on the database schematic changes of a multi-tiered software application and suggest a new technique to detect the faults early that are related to database schematic changes. Our approach is based on an improved technique for the selection and prioritization of the test cases. Our new approach results in the early detection of faults related to database schematic changes and reduction of test suite execution time.

Evaluating Applicability of Combinatorial Testing in an Industrial Environment: A Case Study
Elisa Puoskari, Tanja E. J. Vos, Nelly Condori-Fernandez, and Peter M. Kruse
(Sulake, Finland; Universitat Politècnica de València, Spain; Berner & Mattner, Germany)
Case studies for evaluating tools in software engineering are powerful. Although they cannot achieve the scientific rigour of formal experiments, the results can provide sufficient information to help companies judge if a specific technology being evaluated will benefit their organization.
This paper reports on a case study done for evaluating a combinatorial testing tool in a realistic industrial environment with real objects and subjects. The case study has been executed at Sulake, a company that develops social entertainment games and whose main product is Habbo Hotel, a social network community in the shape of an online Hotel that is visited by millions of teenagers every week all around the world. This paper describes the experimental design of the case study together with the results and decisions that Sulake has taken about the further study, adoption and implantation of these type of tools.

Introducing the QCEP-Testing System for Executable Acceptance Test Driven Development of Complex Event Processing Applications
Johannes Weiss, Peter Mandl, and Alexander Schill
(TU Dresden, Germany; Munich University of Applied Sciences, Germany)
In this paper we introduce the open source testing system QCEP-TS for realizing executable acceptance test driven development (EATDD) of complex event processing (CEP) applications. We will motivate the need of EATDD in this domain and argue that state of the art solutions do not address this topic or have substantial weaknesses. We show the design approach of our testing system including a vendor independent testing syntax. We also state relevant challenges for testing CEP applications and discuss implementation details of our solution. On the basis of a real world scenario we demonstrate first practical experiences of our solution. QCEP-TS is available as open source project at http://sourceforge.net/p/qcep-ts. QCEP (Quality Assurance of Complex Event Processing) is a collaboration project of the Munich University of Applied Sciences, the Technical University of Dresden and the UniCredit Business Integrated Solutions S.C.p.A.

Practical Experience and Evaluation of Continuous Code Static Analysis with C++Test
Vincenzo Ciriello, Gabriella Carrozza, and Stefano Rosati
(SESM, Italy; Selex ES, Italy)
The static code analysis tools are a fundamental instruments to validate the developed code. They allow detecting bugs (as memory leak, accessing arrays out of bounds, etc.. ), structural errors and preventing entire classes of errors. This work refers to the Parasoft's tool "C++test". It helps developers prevent and eliminate defects using rules tuned to nd code patterns that lead to reliability, performance, and security problems. To be practical, the static analysis must be seamlessly in- tegrated into the team's work- ow. The "`continuous static analysis"' allows executing automatically the analysis when the new code is released. In this way is possible to realize a further automation in the software development executing the tool during the night or weekend. The time saved to run the tool can be used to analyze and correct the bugs. When you deal with static analysis, it's important to under- line the problem of false positive. It would be attractive if we could develop a tool that could intercept all defects in a given piece of software with certainty. Not all real errors can always be caught, and not all errors caught can always be real. This work presents a quantitative evaluation on the percentile of false positive generated by C++test.

Model-Based Testing

Model-Based Test Case Generation using Symbolic Execution
Kazuki Munakata, Susumu Tokumoto, and Tadahiro Uehara
(Fujitsu Labs, Japan)
In this paper, we present a test case generation method in which test cases are generated from Excel-based functional specications, called error-check/update specications. This method has the following two characteristics. 1) Logical structures and constraint conditions in error-check/update specications are translated into Java code. Then the trans- lated Java code is executed symbolically. A solution found by solving combinations of the constraint conditions, during the symbolic execution can serve as a test case for the Excel- based specication. 2)We present test case selection method to extract a suitable test-suite from the large number of test cases typically generated by the above symbolic execution. Our test case selection is based on a path condition-based test case selection criterion, proposed in our previous work, in which the selected test cases covers all the atomic conditions of path conditions collected during symbolic execution. In this paper, we extend this method with classication of test cases and priority of variable values with the aim of improv- ing the quality of the selected test-suite. We implemented the test case generation method in our tool and evaluated this method with Excel-based test specications for a real- world nancial system.

Analysis and Testing of Matlab Simulink Models: A Systematic Mapping Study
Frank Elberzhager, Alla Rosbach, and Thomas Bauer
(Fraunhofer IESE, Germany)
Matlab Simulink is one of the major modeling and simulation tools applied in different embedded systems domain. Quality assurance is an essential, but often highly effort-consuming part of software development. A lot of different quality assurance techniques exist to ensure high quality, but these analysis and testing techniques are often applied in isolation. Therefore, we are interested in synergy effects when applying them in combination. Consequently, we performed a systematic mapping study to identify the current state of the art regarding such quality assurance techniques and existing combinations. Our main result is a classification of existing quality assurance techniques applied on Matlab Simulink models, and an overview of existing tool support and the validity of the approaches.

Model-Based Testing in Legacy Software Modernization: An Experience Report
Marc-Florian Wendland, Marco Kranz, Christian Hein, Tom Ritter, and Ana García Flaquer
(Fraunhofer FOKUS, Germany; DOME Consulting & Solutions, Spain)
With the advent of cloud computing more and more vendors strive to modernize legacy applications and deploy them into the cloud. In particular when the legacy system is still applied in the field, the vendor must ensure a seamless change to the modernized system to not lose any economical assets and to keep the business running. As with normal development processes, testing is also inevitable for a modernization process to gain confidence that the modernized system behaves correctly. This paper describes an experience report from the FP 7 research project REMICS that deals with model-driven modernization of legacy systems to the cloud. We employed a model-based testing process for safeguarding the correct migration of the modernized system’s functionality. As test modeling language, the UML Testing Profile was applied. The modernized system, called DOME, was one of the case studies contributed by one of the business partners of the project.

Towards Quality of Model-Based Testing in the ioco Framework
Michele Volpato and Jan Tretmans
(Radboud Universiteit Nijmegen, Netherlands; TNO, Netherlands)
Since testing is an expensive process, automatic testing with smart test selection has been proposed as a way to reduce such expense. Such a selection of tests can be done using specification coverage functions. Model-based ioco theory, however, uses test suites which are not suitable for easy computation of coverage because of interdependence of their test cases. We define a new test suite that overcomes such problems. Using such a test suite we cast the test selection problem to a specification selection problem that aims at transforming the model to which a system under test must conform, in order to reduce the set of test cases. We give a canonical representation for the newly defined test suite.

An Auto-review Tool for Model-Based Testing of Safety-Critical Systems
Atit Mishra, Manjunatha Rao, Chethan CU, Vanishree Rao, Yogananda Jeppu, and Nagaraj Murthy
(Moog India Technology Center, India)
Verification of Test Procedures and Test Cases (TP, TC) is an important process stage for fulfilling the RTCA/DO-178B/C compliance objective while developing software for safety critical flight control systems. Correctness of these results, their requirement coverage (High/Low level) analysis and justified discrepancies (if any) are the primary objectives that must be satisfied. Manual reviews are the most common methods for achieving these objectives. This paper consolidates some of the past experiences of conducting manual reviews in the flight control software V&V for a commercial aircraft program. Through them a clear need and scope for automating portions of manual review is identified. This led to the development of an Auto Review Tool (ART) which is qualified for claiming the certification credits. All the necessary documents such as Tool’s Operational Requirement (TOR), Tool Qualification Plan (TQP) and Tool Assessment Summary (TAS) are prepared. The paper presents the tool development process beginning with defining a library of Input/Output relation based functions for different control blocks, parser development and qualification activity that were performed by an independent testing team.

Test Automation Tools

Comparing the Maintainability of Selenium WebDriver Test Suites Employing Different Locators: A Case Study
Maurizio Leotta, Diego Clerissi, Filippo Ricca, and Cristiano Spadaro
(Università di Genova, Italy; eXact learning solutions, Italy)
Test suite maintenance tends to have the biggest impact on the overall cost of test automation. Frequently modifications applied on a web application lead to have one or more test cases broken and repairing the test suite is a time-consuming and expensive task.
This paper reports on an industrial case study conducted in a small Italian company investigating on the analysis of the effort to repair web test suites implemented using different UI locators (e.g., Identifiers and XPath).
The results of our case study indicate that ID locators used in conjunction with LinkText is the best solution among the considered ones in terms of time required (and LOCs to modify) to repair the test suite to the new release of the application.

Automatic, Load-Independent Detection of Performance Regressions by Transaction Profiles
Shadi Ghaith, Miao Wang, Philip Perry, and John Murphy
(University College Dublin, Ireland)
Performance regression testing is an important step in the production process of enterprise applications. Yet, analysing this type of testing data is mainly conducted manually and depends on the load applied during the test. To ease such a manual task we present an automated, load-independent technique to detect performance regression anomalies based on the analysis of performance testing data using a concept known as Transaction Profile. The approach can be automated and it utilises data already available to the performance testing along with the queueing network model of the testing system.
The presented ``Transaction Profile Run Report'' was able to automatically catch performance regression anomalies ca-used by software changes and isolate them from those caused by load variations with a precision of 80% in a case study conducted against an open source application. Hence, by deploying our system, the testing teams are able to detect performance regression anomalies by avoiding the manual approach and eliminating the need to do extra runs with varying load.

A System for Automated Testing in Development of Measuring Devices for Industrial Process Instrumentation
René Keimling, Christian Hansen, and Attila Bilgic
(KROHNE Messtechnik, Germany)
In this paper, we present the KROHNE TestCenter (KTC), a custom-made test system for automated verification on integration (and partly system) level in the field of industrial process instrumentation. The KTC has been designed and developed at KROHNE to meet demands of (a) aligned test activities amongst various product groups and demands of (b) normative requirements from international standards and industry expectations as well as to account for (c) specific boundary conditions from (partly proprietary) technological solutions. Providing high reproducibility, the developed system allows for test case implementation, remote control of the device under test along with the test equipment, automation of test execution and corresponding analysis as well as an automated generation of documentation. The system has been utilized in multiple running development projects, on the one hand contributing to test activities and on the other hand feeding back gained experiences in enhancements of KTC. This paper reflects initial results obtained over the past 1.5 years. It is envisaged to extend the system and use it as a standard tool within the KROHNE group.

FITNESS: A Framework for Automatic Testing of ASTERIX Based Software Systems
Vittorio Manetti and Luigi Martin Petrella
(SESM, Italy)
As applications are developed, functional tests ensure they continue to function as expected. Nowadays, functional testing is mostly done manually, with human testers verifying a systems functionality themselves, following hand-written instructions: this make testing of software components one of the most expensive phases in the software development cycle, either in terms of time as well as human effort. Concerning in particular safety critical systems, such as the ones belonging to the Air Traffic Management field, for which it is always necessary to be taken complete and rigorous security test and evaluation among development team and/or by third-party security certification organization, performing automatic tests on such systems become a very tricky process considering that the goal is to verify not only the proper functioning of the SUT, but the system dependability too. However, such software testing is usually time consuming, cost consuming and boresome and thus technologies of software testing automation have alluring application foreground in that field: making the execution of test cases automatic allows to reduce costs and to improve software quality from a dependability point of view. In this paper we present FITNESS, a framework for the automation of testing procedures for complex software systems with strict safety and quality requirements, and in particular we have focused on Air Traffic Control (ATC) application who rely on ASTERIX standard as data exchange format with the intent to propose a flexible solution to automate testing procedure for a generic system that use such communication standard. We also present a quantitative study that analyze the effectiveness of the proposed approach using our framework to test a Secondary Surveillance Radar system and showing that most of manual test steps can be automatically converted to automated test steps with no human intervention.

proc time: 0.69