PLATEAU 2015 – Proceedings

Foreword
Welcome to the proceedings of the 6th Workshop on Evaluation and Usability of Programming Languages and Tools (PLATEAU), Pittsburgh, PA, USA, October 26, 2015, co-located with SPLASH 2015. This year’s workshop continues its tradition of being the forum for presentation of research results and experience re- ports on methods, metrics, and techniques for evaluating the usability of languages and language tools. PLATEAU gives researchers and practitioners a unique opportunity to share their perspectives with others interested in the various aspects of evaluation of programming languages and tools.

Tools

Frequency Distribution of Error Messages
David Pritchard
(University of Waterloo, Canada)
Which programming error messages are the most common? We investigate this question, motivated by writing error explanations for novices. We consider large data sets in Python and Java that include both syntax and run-time errors. In both data sets, after grouping essentially identical messages, the error message frequencies empirically resemble Zipf-Mandelbrot distributions. We use a maximum-likelihood approach to fit the distribution parameters. This gives one possible way to contrast languages or compilers quantitatively.

Info

An Evaluation of the DiaSuite Toolset by Professional Developers: Learning Cost and Usability
Milan Kabáč, Nic Volanschi, and Charles Consel
(INRIA, France; University of Bordeaux, France)
This paper evaluates a design-driven, tool-based approach, named DiaSuite, dedicated to developing applications involving sensors and actuators. Specifically, we evaluate the usability and the learning cost of DiaSuite as a first step to assess the potential for transferring this technology to the industrial practice of this domain.
We assess the cost of learning DiaSuite by involving four professional programmers in a usability study involving a software engineering task. This experiment brings preliminary evidence that the DiaSuite technology can be used effectively by professional developers after only half a day of training.
We then present qualitative data about the usage and usability of DiaSuite, collected from developers, via questionnaires and interviews. Finally, we discuss lessons learned from this work.

Aiding Programmers using Lightweight Integrated Code Visualization
Per Ola Kristensson and Chung Leung Lam
(University of Cambridge, UK)
We present a Lightweight Integrated Code Visualization (LICV) tool designed to aid programmers using Integrated Development Environments (IDEs). LICV is implemented as a plug-in for the Eclipse IDE for Java Developers. LICV continuously tracks the active editor in the IDE and visualizes up to 24 code features in a designated non-intrusive view. LICV is designed to facilitate fast understanding of the structure of the code in order to help users carry out routine programming tasks. It enables users to zoom, filter, search, and go back and forth between the code and the visualization via direct manipulation. We evaluated LICV by carrying out two user studies which compared LICV against regular Eclipse in four tasks. We found that LICV significantly reduced participants’ completion times by nearly 50% for three out of four tasks. Further, participants significantly preferred using LICV to perform the tasks.

Towards Moldable Development Tools
Andrei Chiş, Oscar Nierstrasz, and Tudor Gîrba
(University of Bern, Switzerland; tudorgirba.com, Switzerland)
Developers commonly ask detailed and domain-specific questions about the software systems they are developing and maintaining. Integrated development environments (IDEs) form an essential category of tools for developing software that should support software engineering decision making. Unfortunately, rigid and generic IDEs that focus on low-level programming tasks, that promote code rather than data, and that suppress customization, offer limited support for informed decision making during software development. We propose to improve decision making within IDEs by moving from generic to context-aware IDEs through moldable tools. In this paper, we promote the idea of moldable tools, illustrate it with concrete examples, and discuss future research directions.

Info

Understanding the Effects of Code Presentation
Jason T. Jacques and Per Ola Kristensson
(University of Cambridge, UK)
The majority of software is still written using text-based programming languages. With today's large, high-resolution color displays, developers have devised their own "folk design" methodologies to exploit these advances. As software becomes more and more critical to everyday life, supporting developers in rapidly producing and revising code accurately should be a priority. We consider how layout, typefaces, anti-aliasing, syntax highlighting, and semantic highlighting might impact developer efficiency and accuracy.

Programming Languages

A User Study for Comparing the Programming Efficiency of Modifying Executable Multimodal Interaction Descriptions: A Domain-Specific Language versus Equivalent Event-Callback Code
Fredy Cuenca, Jan Van den Bergh, Kris Luyten, and Karin Coninx
(University of Hasselt, Belgium; Transnational University of Limburg, Belgium; iMinds, Belgium)
The present paper describes an empirical user study intended to compare the programming efficiency of our proposed domain-specific language versus a mainstream event language when it comes to modify multimodal interactions. By concerted use of observations, interviews, and standardized questionnaires, we managed to measure the completion rates, completion time, code testing effort, and perceived difficulty of the programming tasks along with the perceived usability and perceived learnability of the tool supporting our proposed language. Based on this experience, we propose some guidelines for designing comparative user studies of programming languages. The paper also discusses the considerations we took into account when designing a multimodal interaction description language that intends to be well regarded by its users.

A Study on the Most Popular Questions about Concurrent Programming
Gustavo Pinto, Weslley Torres, and Fernando Castor
(Federal University of Pernambuco, Brazil)
Concurrent programming is notably known as a hard discipline. Over the last few years, great strides have been made in improving concurrent programming abstractions, techniques, and tools to ease concurrent programming practice. However, little effort has been placed on assessing what are the real-world problems faced by developers when writing concurrent applications. In this paper, we describe an empirical investigation of the top-250 most popular questions about concurrent programming on StackOverflow. We observed that even though some questions (22.94%) are related to practical problems (e.g., “how to fix this concurrency bug”), most of them (66.23%) are related to basic concepts (e.g., “what is a mutex?”), which were created by well- experienced StackOverflow users. Curiously, we did not find any question about how to use concurrent programming techniques to improve application performance.

Comparing Transitive to Non-transitive Object Immutability
Michael Coblenz, Joshua Sunshine, Brad A. Myers, Sam Weber, and Forrest Shull
(Carnegie Mellon University, USA; Software Engineering Institute, USA)
Many programming languages provide features that express restrictions on which data structures can be changed. For example, C++ includes const and Java includes final. Languages that are in widespread use typically provide non-transitive immutability: when a reference is specified to be immutable or read-only, the object referenced can still reference mutable structures. However, some languages, particularly research languages, provide transitive immutability, in which immutable objects can only reference other immutable objects (with some exceptions). We are designing a lab study of programmers to elucidate the differences in programmer effectiveness between these two approaches.

Is Functional Programming Better for Modularity?
Ismael Figueroa and Romain Robbes
(Pontificia Universidad Católica de Valparaíso, Chile; University of Chile, Chile)
In 1989 John Hughes published an influential position paper entitled Why Functional Programming Matters. The article extolls the virtues of lazy functional programming by developing several examples: the Newton-Rhapson squares root method, numerical differentiation and integration, and an alpha-beta minimax search. A main conclusion of that work is that higher-order functions and lazy evaluation significantly contribute to modularity. We have found that recent articles from 2010 to 2014 cite Hughes' work as seminal work supporting that functional programming is, in general, good for modularity. We believe this reflects an unstated hypothesis in part of the research community: functional programming is inherently better at modularity than other paradigms such as typical procedural and object-oriented programming. To the best of our knowledge there are no (large-scale) empirical evaluations of this characteristic. We discuss the influence of Why Functional Programming Matters on current beliefs regarding the advantages of functional programming, the recent citations that intrigues us, and provide a small experiment on the GHC Haskell compiler, suggesting the existence of modularity issues in it.

Operators and Precedence in Programming Languages
Najwani Razali, James Noble, and Stuart Marshall
(Victoria University of Wellington, New Zealand)
An incorrect understanding of operators and precedence can create bugs in programs. The purpose of this study is to explore programmers understanding, interpretation and perception of operators and precedence issues. Parentheses have the highest precedence in programming languages. So, the important of parentheses in expressions will be tested. The findings may lead to an improvement to rules for operators and precedence in programming languages. The findings will also provide both theoretical analyses and guidelines for developers and language designers.

Some Usability Hypotheses for Verification
David J. Pearce
(Victoria University of Wellington, New Zealand)
The idea of specifying and verifying software to eliminate errors has been studied extensively over the last three decades or more. Recent advances in automated theorem proving have given rise to a range of new verification tools being developed. Despite this, very little is known about the effect of using such tools on software development. In this paper, we present several verification-related usability hypotheses which we believe warrant further investigation. These hypotheses are based on observations from the literature, as well as our own experiences in developing and using the Whiley verification system.

Group Discussion

On the Need to Define Community Agreements for Controlled Experiments with Human Subjects: A Discussion Paper
Stefan Hanenberg and Andreas Stefik
(University of Duisburg-Essen, Germany; University of Nevada at Las Vegas, USA)
While it looks like controlled trials with human involvement are increasingly applied in software science, there are few explicitly documented community standards in regard to their design or approach. This leads to a number of problems: experimenters cannot be sure whether an experiment they perform does represent the current state-of-the-art, reviewers have no guidelines to check whether a critique they have in mind is valid or not, and readers from experiments have hardly any chance to check whether the results of an experiment they are reading should be taken seriously. This paper discusses the problem of missing community standards for empirical studies in computer science and makes a first proposal with respect to subjects, training, measurements, experimental designs, and documentation. The overall goal of this paper is to begin a discussion on this issue.

PLATEAU 2015 – Proceedings

Frontmatter

Tools

Programming Languages

Group Discussion