ICSE 2013 Workshops
2013 35th International Conference on Software Engineering (ICSE)
Powered by
Conference Publishing Consulting

2013 1st International Workshop on Conducting Empirical Studies in Industry (CESI), May 20, 2013, San Francisco, CA, USA

CESI 2013 – Proceedings

Contents - Abstracts - Authors

1st International Workshop on Conducting Empirical Studies in Industry (CESI)


Title Page

It is our pleasure to welcome the reader into the (pre-workshop) proceedings of the 1st International Workshop on Conducting Empirical Studies in Industry (CESI 2013) held in conjunction with the 35th International Conference on Software Engineering (ICSE 2013). The discipline of Software Engineering (SE) has come a long way from the era of crisis from which it was born circa 1968. We are at a point where we recognise that empirical research is an important driver for moving the SE field forward. In response to this, it is to be expected that more and more researchers are conducting empirical research in SE. With increased maturity, however, comes increased responsibility.


Parametric Model CESI Experience (Keynote)
Barry Boehm
(University of Southern California, USA)
This presentation will summarize four decades of conducting empirical studies in industry in the area of parametric estimation models. It will identify several critical success factors, and will compare them to those of other types of empirical studies. Finally, synergies about different forms of empirical studies will be discussed.

Article Search

Regular Papers

Software Industry Experiments: A Systematic Literature Review
Oscar Dieste, Natalia Juristo, and Mauro Danilo Martínez
(UPM, Spain; University of Oulu, Finland; Escuela Politécnica del Ejército, Ecuador)
Background: There is no specialized survey of experiments conducted in the software industry. Goal: Identify the major features of software industry experiments, such as time distribution, independent and dependent variables, subject types, design types and challenges. Method: Systematic literature review, taking the form of a scoping study. Results: We have identified 10 experiments and five quasi-experiments up to July 2012. Most were run as of 2003. The main features of these studies are that they test technologies related to quality and management and analyse outcomes related to effectiveness and effort. Most experiments have a factorial design. The major challenges faced by experimenters are to minimize the cost of running the experiment for the company and to schedule the experiment so as not to interfere with production processes. Conclusion: Companies appear to be disinclined to run experiments because they are not perceived to have direct benefits. We believe that researchers staging a field experiment in a company should adopt a business-aligned stance and plan an experiment that clearly benefits managers and professionals.

Article Search
Conducting Empirical Studies in Industry: Balancing Rigor and Relevance
Shilpi Jain, Muhammad Ali Babar, and Jude Fernandez
(Infosys, India; Lancaster University, UK; IT University of Copenhagen, Denmark)
Empirical Software Engineering research has achieved considerable results in building our knowledge about selecting and applying appropriate empirical methods for technology evaluation. Empirical studies in general and empirical studies in industrial settings in particular have played an important role in successful transition of many Software Engineering technologies to industry, for example, defect detection techniques and automated test cases. However, conducting empirical research in industrial settings remains a challenging undertaking for a variety of reasons. There is no substantial literature reporting on the challenges and complexities involved in conducting empirical studies in an industry in general and in settings whose business models are built around global sourcing. This paper reports some of our experiences and lessons learned from conducting empirical research in industry. Some of the observed challenges include short time horizon for research, high expectations, limited research skills, and the acceptable research rigor. The paper discusses some of these issues with relevant examples and provides some strategies for overcoming these issues. We also stress that researchers and practitioners should share their experiences of conducting empirical research in order to help build a body of knowledge to guide the future efforts.

Article Search
Industrial Study on Test Driven Development: Challenges and Experience
Adnan Čaušević, Rakesh Shukla, and Sasikumar Punnekkat
(Mälardalen University, Sweden; Infosys, India)
Conducting empirical studies in industry always presents a major challenge for many researchers. Being a graduate student does not make things any easier. Often due to the lack of experience, credibility or just very limited networking, graduate students do not receive many opportunities to directly collaborate with industry and experiment their theoretical models in a realistic environment. On the other hand, empirical research conducted in an academic settings is often criticised for using students as subjects and working with a small sample size, thus creating major validity threat of the published results. In this paper we are presenting an experience report from an industrial empirical study conducted at Infosys Ltd., India with the support of their global internship program for graduate students, InStep. Focus of the paper is to present several challenges arisen before, during, and after the study, requiring an immediate attention in order to have a successful experiment completion. We also discuss and elaborate the data analysis results and its implication to our current research activities.

Article Search
Six Reasons for Rejecting an Industrial Survey Paper
Marco Torchiano and Filippo Ricca
(Politecnico di Torino, Italy; University of Genova, Italy)
Despite their importance in any empirically based research program, industrial surveys are not very common in the software engineering literature. In our experience, a possible reason is their difficulty of publication. We would like to understand what are the issues that may prevent the publication of papers reporting industrial surveys. In this preliminary work, we analyzed the surveys we conducted and extracted the main lessons learned in terms of issues and problems. Most common critics posed to industrial surveys are: lack of novelty, limitation of the geographic scope and sampling issues. Most objections that led to reject a survey paper actually are not easy to overcome and others are not so serious. These objections could restrain researchers from conducting this type of studies that represent an important methodological asset. For these reasons, we think that reviewers should be less severe to judge survey papers provided that all the limitations of the study are well explained and highlighted.

Article Search
Success Factors for Empirical Studies in Industry-Academia Collaboration: A Reflection
Paul Grünbacher and Rick Rabiser
(JKU Linz, Austria)
We conducted several empirical studies over the last couple of years as part of a collaboration with industrial partners in the area of software product lines. Our studies differed regarding their motivation and goals, their scope, the research methods applied, and the involved subjects. In this experience paper we briefly summarize the studies and their key characteristics. We reflect on our experiences based on the success factors for industry-academia collaborations by Wohlin et al. We discuss the role of empirical studies in long-term industry-academia collaborations and present lessons learned.

Article Search
Mind the Gap and Find Common Ground: Empirical Research in Multiple Firms
Naomi Unkelos-Shpigel and Irit Hadar
(University of Haifa, Israel)
The contribution of academic research to projects in industry has been discussed and highlighted; yet it is still not easy for an academic researcher to establish collaboration with industrial partners. In this paper we describe an empirical research conducted in collaboration with multiple firms. This research was originally initiated as a single case study in collaboration with one firm, and gradually evolved to be an interpretive study in seven global firms. This transition helped us to overcome challenges that emerged while we were conducting a qualitative research based on a single case study. In this paper we present the reasons that led us to expend beyond the original case study, and discuss the benefits as well as the pitfalls of collecting empirical data, and specifically conducting interviews, in multiple firms.

Article Search
Generalizing by Similarity: Lessons Learnt from Industrial Case Studies
Smita Ghaisas, Preethu Rose, Maya Daneva, Klaas Sikkel, and Roel J. Wieringa
(Tata Consultancy Services, India; University of Twente, Netherlands)
Large IT vendors execute thousands of projects in a variety of business domains and environments. Over the years, they end up repeatedly developing and deploying systems for a given domain in the same country, sometimes even for the same company. It would save them a lot of cost and effort if they could reliably depend on their past experience and draw insights from lessons learnt in the past. However, this requires them to generalize from past projects to a similar current project. In this paper we draw lessons learnt from three industrial case studies on how to generalize by similarity.

Article Search

Short Papers

Empirical Software Engineering Research with Industry: Top 10 Challenges
Claes Wohlin
(Blekinge Institute of Technology, Sweden)
Software engineering research can be done in many ways, in particular it can be done in different ways when it comes to working with industry. This paper presents a list of top 10 challenges to work with industry based on our experience from working with industry in a very close collaboration with continuous exchange of knowledge and information. The top 10 list is based on a large number of research projects and empirical studies conducted with industrial research partners since 1983. It is concluded that close collaboration is a long-term undertaking and a large investment. The importance of addressing the top 10 challenges is stressed, since they form the basis for a long-term sustainable and successful collaboration between industry and academia.

Article Search
Conducting a Long-Term Case Study in a Software Firm: An Experience Report
Sofia Sherman and Irit Hadar
(University of Haifa, Israel)
Academia and industry have been collaborating for over a century in different formats, including research projects funds, commercialization of academic knowledge, academic consultancy in industrial projects, etc. Yet, academic researchers face various difficulties and challenges when performing research in industry. In this paper we describe several challenges we faced during a long-term case study research we performed in collaboration with a global IT firm. We present each challenge including its illustration in the context of our case study, describe the actions taken in the attempt to meet it, and discuss lessons learned.

Article Search
Conducting Interview Studies: Challenges, Lessons Learned, and Open Questions
Jim Witschey, Emerson Murphy-Hill, and Shundan Xiao
(North Carolina State University, USA)
Our recent work uses sociological theories and interview techniques to discover why so few developers use tools that help them write secure code. In this experience report, we describe nine challenges we encountered in planning and conducting an interview study with industrial practitioners, from choosing a population of interest to presenting the work in a way that resonates with the research community. In doing so, we aim to spur discussion in the software engineering research community about common challenges in empirical research and ways to address them.

Article Search
Confounding Factors When Conducting Industrial Replications in Requirements Engineering
David Callele, Krzysztof Wnuk, and Markus Borg
(TR Labs, Canada; Lund University, Sweden)
Despite the widely recognized importance of replications in software engineering, industrial replications in software engineering are still rarely reported. Although the literature provides some evidence about the issues and challenges related to conducting experiments and replications the practitioners view of the issues and challenges has not been fully explored. This paper reports an industrial practitioners review of a replicated experiment on linguistic tool support for consolidation of requirements from multiple sources. The review identified potential confounding factors from a perspective that differed significantly from that of the designers of the experiment. The results suggest that industrial practice may focus upon specific process aspects that are not necessarily reflected in academic practice.

Article Search
Plat_Forms: Contests as an Alternative Approach to SE Empirical Studies in Industry
Lutz Prechelt and Ulrich Stärk
(Freie Universität Berlin, Germany)
Plat_Forms is a set of three studies that search for process and product properties emerging from the use of specific web development platforms. Its first key property is the use of a contest format; this leads to easier, broader, and more uniform data collection compared to on-site industrial studies. As a second key property, the three instances do not merely serve to extend or corroborate each others' results, they also serve as a time series for characterizing changes in the underlying software development reality over the course of a few years -- an important aspect that has received little attention in empirical software engineering research so far.

Article Search
Lessons Learned from Conducting Industry Surveys in Software Testing
Tanjila Kanij, Robert Merkel, and John Grundy
(Swinburne University of Technology, Australia; Monash University, Australia)
We have conducted a number of industry studies looking into aspects of software testing. These include a survey of practitioner view on software tester performance indicators and team formation issues; detailed worklog analysis of representative tasks conducted by software testers; feedback on a proposed approach to software tester performance evaluation; and collecting indicative personality factors of software testers vs other software developers. These have proved to be challenging exercises. Key issues to overcome include reaching appropriate testers in industry, gaining management and company consent, structuring surveys and questionnaires to balance time vs detail, and gaining sufficient response rate and response quality to be useful.

Article Search

Practitioner Messages

Can We Do Useful Industrial Software Engineering Research in the Shadow of Lean and Agile?
Kristian Wiklund, Sigrid Eldh, Daniel Sundmark, and Kristina Lundqvist
(Ericsson, Sweden; Mälardalen University, Sweden)
The software industry is rapidly changing from traditional ways of working to lean and agile development methods using self-organized feature development teams that are performing a much larger part of the development process than before. Face to face communication will replace many of the design artifacts used for work-in-progress, such as defect reports and feature system design specifications. This type of data will cease to exist when the feature is developed or the problem is solved, and will not be readily available to researchers. As a consequence, software engineering research in industry will have to rely primarily on participatory and observational methods.

Article Search
Some Researcher Considerations When Conducting Empirical Studies in Industry
Sigrid Eldh
(Ericsson, Sweden; Karlstad University, Sweden)
Research collaborations must be set up for a win-win relationship, avoiding many hurdles along the way. In this short paper the goal is to identify some considerations when conducting empirical studies in industry. These considerations deal with problem solution approaches, communication focus, unpleasant results, mismatching interpretations and scope, business aspects and some evaluation criteria for a successful collaboration.

Article Search

proc time: 0.49