Powered by
Conference Publishing Consulting

2nd International Workshop on Conducting Empirical Studies in Industry (CESI 2014), June 2, 2014, Hyderabad, India

CESI 2014 – Proceedings

Contents - Abstracts - Authors

2nd International Workshop on Conducting Empirical Studies in Industry (CESI 2014)

Title Page


Message from the Chairs
While there are established research methods in the methodology literature, relatively little is known about conducting empirical studies involving the software or IT industry. How to ensure that what is being investigated are the appropriate constructs; what pitfalls to avoid when investigating phenomena in an organisation; what challenges to anticipate when evaluating the efficacy of methods and tools in actual projects; how to ensure continued interest from the industry stakeholders and participants in a study; what are the dos and don’ts when conducting practitioner surveys; etc. Such questions abound and they formed the primary trigger for organising this workshop for the second consecutive year, after the success of the first workshop held at ICSE 2013.

A Career Spent Wading through Industry's Empirical Ooze (Invited Talk)
Bill Curtis
(CAST Software, USA)
Sometimes analyzing industrial software data seems like draining a swamp. This keynote describes challenges faced while wading through 5 layers of empirical ooze over 36+ years in the software industry. The first layer seeped from formal experiments. The second layer spilled from a global corporate repository of project benchmark data. The third layer leaked from verbalizations by designers and design teams. The fourth layer was emitted by case studies of process improvement. The fifth layer dripped from structural quality data on large IT systems. Five lessons for separating the ooze from the data will be recounted.

Is the Glass Half Empty or Half Full? Conducting Empirical Studies in Industry (Invited Talk)
Ipek Ozkaya
(SEI, USA)
Building effective collaborations, scoping a research problem to produce results that will benefit the industry, and lacking access to data are often listed as barriers to conducting academic studies in industry. Despite these challenges, research that uses empirical approaches and that collaborates with industry is increasing; there are even conferences and journals dedicated to the cause. The software engineering research community aims to produce results that are valuable to practice through these endeavors. Yet, one key goal of this research seems to get lost as we try to sort through the challenges that such collaborations impose upon the work plan: Do the results of these studies really help us innovate and produce research outputs that others can use, build on, and replicate? Or are we becoming victims of the available empirical software engineering methods? In this presentation, I will give examples of how the results of empirical studies have helped shape research problems in which industry can more easily take part and even be engaged partners in the work.

Exploring Web Advertising to Attract Industry Professionals for Software Engineering Surveys
Matthias Galster and Dan Tofan
(University of Canterbury, New Zealand; University of Groningen, Netherlands)
The validity of questionnaire-based surveys in empirical software engineering studies depends on the number of survey participants and the quality of the data collected from participants. Attracting respondents is a continuous challenge and many studies rely on participants from the personal network of researchers. In this short paper we present several insights from using web advertising to recruit participants for one of our surveys on software architecture decision making. More specifically, in this work we show how we used web advertising in addition to contacting our personal networks, postings on mailing lists, blogs, etc. to reach a larger and broader base of our target population. We discuss the lessons we learned through this experience, and potentials and trade-offs of using web advertising in software engineering surveys.

Lessons Learned on Applying Design Science for Bridging the Collaboration Gap between Industry and Academia in Empirical Software Engineering
Pilar Rodríguez, Pasi Kuvaja, and Markku Oivo
(University of Oulu, Finland)
Collaboration between industry practitioners and researchers has been reported as a challenge to conduct empirical studies in software engineering. Gaining practitioners interest in research activities is challenging in the software development industry characterized by tough competition and short time-to-market. Often, practitioners do not see the direct value in collaborating with researchers. Meanwhile, researchers are frequently frustrated because they have no access to the real-world data that would enable them to develop scientific knowledge. Design science has become a popular research framework in Information Systems. This paper describes our lessons learned on how design science can help to attract and motivate industry practitioners to take part in empirical research. We present our practical experiences with using design science in the context of empirical software engineering, and how the design science framework helped us to bridge the collaborative gap with our industry partners. In illustrating our experiences, we present two case studies conducted in collaboration with Ericsson and Elektrobit in which the design science framework was followed. Based on our experience, design science is a suitable research framework for empirical software engineering since much of the research in the field is constructive. The similarities between the design science practices and the day-to-day work of practitioners in industry make it easy for them to understand the research approach. Thus, it supports their willingness to participate in research activities.

Practical Experiences in Designing and Conducting Empirical Studies in Industry-Academia Collaboration
Silverio Martínez-Fernández and Helena Martins Marques
(Universitat Politècnica de Catalunya, Spain; everis, Spain)
More and more, software engineering researchers are motivated to solve real problems that bring value to industry. An example is the industry-academia collaboration described in this paper among everis, an IT consulting firm, and the GESSI research group at UPC. The goal of this paper is twofold: to evaluate the success of the collaboration, and to report the experience with conducting empirical studies in the industry and lessons learnt. We evaluated our collaboration with an existing model for technology transfer, and performed a focus group discussion to identify challenges we have faced. After initialization and alignment of the collaboration, a high maturity level has been achieved: we have obtained the first results in form of proposed solutions, scientific publications, and pilots run in real projects. In spite of this positive progress, further initiatives need to be undertaken in the last phases of the collaboration to achieve high degrees of maturity in deployment impact, industry benefit and innovativeness. Evaluating the collaboration has been positive, since we identified the next steps to be taken to achieve a high degree of technology transfer and innovation dissemination. We think it is a needed step in industry-academia collaborations in order to improve their success.

Empirical Evaluation of Software Testing Techniques in an Open Source Fashion
Sheikh Umar Farooq and SMK Quadri
(University of Kashmir, India)
Testing technique selection and evaluation remains a key issue in software testing. Industry practitioners need concrete evidence to select proper testing techniques in STLC. Despite the large number of empirical studies which attempt to study the testing techniques’ applicability conditions and allied factors, we are still without realistic and generalized results as studies lack a formal foundation and are not complete in all respects. Additionally, besides varying significantly in terms of parameters they have taken into consideration, many existing studies show contradictory results. Even though the researchers stress on replication of these studies under a common set of guidelines, however, attempts to aggregate results from such replications still has not been fruitful so far. As such, to bridge the gap between researchers and industry professionals, we propose to carry out evaluation of testing techniques on a large scale under a unified framework in an open-source fashion so that the realistic and generalized results are obtained in a shorter span of time.

Topic Selection in Industry Experiments
Ayse Tosun Misirli, Hakan Erdogmus, Natalia Juristo, and Oscar Dieste
(University of Oulu, Finland; Carnegie Mellon University, USA; Universidad Politécnica de Madrid, Spain)
This paper shares our experience with initial negotiation and topic elicitation process for conducting industry experiments in six software development organizations in Finland. The process involved interaction with company representatives in the form of both multiple group discussions and separate face-to-face meetings. Fitness criteria developed by researchers were applied to the list of generated topics to decide on a common topic. The challenges we faced include diversity of proposed topics, communication gaps, skepticism about research methods, initial disconnect between research and industry needs, and lack of prior work relationship. Lessons learned include having enough time to establish trust with partners, importance of leveraging the benefits of training and skill development that are inherent in the experimental approach, uniquely positioning the experimental approach within the landscape of other validation approaches more familiar to industrial partners, and introducing the fitness criteria early in the process.

On the Relationship between Quality Assurance and Productivity in Software Companies
Carlos Henrique C. Duarte
(BNDES, Brazil)
Quality assurance methods based on software process im- provement models have been regarded as a main source of variability in software productivity. In this paper, we investi- gate the relationship between labor productivity and quality assurance levels, using a data set containing more than 500 Brazilian software rms. We perform statistical analyses relating labor productivity, as measured through the annual gross revenue per worker ratio, to quality levels, whose ma- turity was examined in appraisals performed from 2006 to 2012 according to two distinct software process improvement models (CMMI and MPS.BR). As a preparatory step to our ndings, we investigate the relationship between these mod- els. We show that CMMI and MPS.BR appraised maturity levels are correlated, but we nd no statistical evidence that the implemented quality assurance methods are related to higher labor productivity or productivity growth.

An Exploratory Study on Improving Automotive Function Specifications
Andreas Vogelsang
(TU München, Germany)
In automotive system development, function specifications describe the requirements and basic design of a function. They are central artifacts and serve as inputs for several activities. With increasing complexity of functions in a vehicle, function specifications become harder to comprehend, change and validate.
For the purpose of increasing our understanding of what the reasons for these impediments are and how they are related in order to derive potential for improvement, we conducted a qualitative (grounded theory) study. In this study, we interviewed nine senior practitioners of an automotive company on how they work with function specifications and which problems they encounter.
In this paper, we show the results of this study and report on experiences and challenges of conducting a grounded theory study in industry.

proc time: 0.91