Programming Journal, Volume '22: "Automated, Targeted Testing ..."
Automated, Targeted Testing of Property-Based Testing Predicates
Tim Nelson, Elijah Rivera, Sam Soucie, Thomas Del Vecchio, John Wrenn, and Shriram Krishnamurthi
(Brown University, USA; Massachusetts Institute of Technology, USA; Indiana University, USA)
Context This work is based on property-based testing (PBT). PBT is an increasingly important form of software testing. Furthermore, it serves as a concrete gateway into the abstract area of formal methods. Specifically, we focus on students learning PBT methods.
Inquiry How well do students do at PBT? Our goal is to assess the quality of the predicates they write as part of PBT. Prior work introduced the idea of decomposing the predicate’s property into a conjunction of independent subproperties. Testing the predicate against each subproperty gives a “semantic” understanding of their performance.
Approach The notion of independence of subproperties both seems intuitive and was an important condition in prior work. First, we show that this condition is overly restrictive and might hide valuable information: it both undercounts errors and makes it hard to capture misconceptions. Second, we introduce two forms of automation, one based on PBT tools and the other on SAT-solving, to enable testing of student predicates. Third, we compare the output of these automated tools against manually-constructed tests. Fourth, we also measure the performance of those tools. Finally, we re-assess student performance reported in prior work.
Knowledge We show the difficulty caused by the independent subproperty requirement. We provide insight into how to use automation effectively to assess PBT predicates. In particular, we discuss the steps we had to take to beat human performance. We also provide insight into how to make the automation work efficiently. Finally, we present a much richer account than prior work of how students did.
Grounding Our methods are grounded in mathematical logic. We also make use of well-understood principles of test generation from more formal specifications. This combination ensures the soundness of our work. We use standard methods to measure performance.
Importance As both educators and programmers, we believe PBT is a valuable tool for students to learn, and its importance will only grow as more developers appreciate its value. Effective teaching requires a clear understanding of student knowledge and progress. Our methods enable a rich and automated analysis of student performance on PBT that yields insight into their understanding and can capture misconceptions. We therefore expect these results to be valuable to educators.
@Article{Programming Journal, Volume22p10,
author = {Tim Nelson and Elijah Rivera and Sam Soucie and Thomas Del Vecchio and John Wrenn and Shriram Krishnamurthi},
title = {Automated, Targeted Testing of Property-Based Testing Predicates},
journal = {},
volume = {},
number = {},
articleno = {10},
numpages = {29},
doi = {10.22152/programming-journal.org/2022/6/10},
year = {2022},
}
Publisher's Version
Info
Programming Journal, Volume '22: "A Transient Semantics for ..."
A Transient Semantics for Typed Racket
Ben Greenman, Lukas Lazarek, Christos Dimoulas, and Matthias Felleisen
(Brown University, USA; Northeastern University, USA; Northwestern University, USA)
Mixed-typed languages enable programmers to link typed and untyped components in various ways. Some offer rich type systems to facilitate the smooth migration of untyped code to the typed world; others merely provide a convenient form of type Dynamic together with a conventional structural type system. Orthogonal to this dimension, Natural systems ensure the integrity of types with a sophisticated contract system, while Transient systems insert simple first-order checks at strategic places within typed code. Furthermore, each method of ensuring type integrity comes with its own blame-assignment strategy.
Typed Racket has a rich migratory type system and enforces the types with a Natural semantics. Reticulated Python has a simple structural type system extended with Dynamic and enforces types with a Transient semantics. While Typed Racket satisfies the most stringent gradual-type soundness properties at a significant performance cost, Reticulated Python seems to limit the performance penalty to a tolerable degree and is nevertheless type sound. This comparison raises the question of whether Transient checking is applicable to and beneficial for a rich migratory type system.
This paper reports on the surprising difficulties of adapting the Transient semantics of Reticulated Python to the rich migratory type system of Typed Racket. The resulting implementation, Shallow Typed Racket, is faster than the standard Deep Typed Racket but only when the Transient blame assignment strategy is disabled. For language designers, this report provides valuable hints on how to equip an existing compiler to support a Transient semantics. For theoreticians, the negative experience with Transient blame calls for a thorough investigation of this strategy.
@Article{Programming Journal, Volume22p9,
author = {Ben Greenman and Lukas Lazarek and Christos Dimoulas and Matthias Felleisen},
title = {A Transient Semantics for Typed Racket},
journal = {},
volume = {},
number = {},
articleno = {9},
numpages = {26},
doi = {10.22152/programming-journal.org/2022/6/9},
year = {2022},
}
Publisher's Version
Programming Journal, Volume '22: "A Transient Semantics for ..."
A Transient Semantics for Typed Racket
Ben Greenman, Lukas Lazarek, Christos Dimoulas, and Matthias Felleisen
(Brown University, USA; Northeastern University, USA; Northwestern University, USA)
Mixed-typed languages enable programmers to link typed and untyped components in various ways. Some offer rich type systems to facilitate the smooth migration of untyped code to the typed world; others merely provide a convenient form of type Dynamic together with a conventional structural type system. Orthogonal to this dimension, Natural systems ensure the integrity of types with a sophisticated contract system, while Transient systems insert simple first-order checks at strategic places within typed code. Furthermore, each method of ensuring type integrity comes with its own blame-assignment strategy.
Typed Racket has a rich migratory type system and enforces the types with a Natural semantics. Reticulated Python has a simple structural type system extended with Dynamic and enforces types with a Transient semantics. While Typed Racket satisfies the most stringent gradual-type soundness properties at a significant performance cost, Reticulated Python seems to limit the performance penalty to a tolerable degree and is nevertheless type sound. This comparison raises the question of whether Transient checking is applicable to and beneficial for a rich migratory type system.
This paper reports on the surprising difficulties of adapting the Transient semantics of Reticulated Python to the rich migratory type system of Typed Racket. The resulting implementation, Shallow Typed Racket, is faster than the standard Deep Typed Racket but only when the Transient blame assignment strategy is disabled. For language designers, this report provides valuable hints on how to equip an existing compiler to support a Transient semantics. For theoreticians, the negative experience with Transient blame calls for a thorough investigation of this strategy.
@Article{Programming Journal, Volume22p9,
author = {Ben Greenman and Lukas Lazarek and Christos Dimoulas and Matthias Felleisen},
title = {A Transient Semantics for Typed Racket},
journal = {},
volume = {},
number = {},
articleno = {9},
numpages = {26},
doi = {10.22152/programming-journal.org/2022/6/9},
year = {2022},
}
Publisher's Version
Programming Journal, Volume '22: "Type Checking Extracted Methods ..."
Type Checking Extracted Methods
Yuquan Fu and Sam Tobin-Hochstadt
(Indiana University, USA)
Many object-oriented dynamic languages allow programmers to extract methods from objects and treat them as functions. This allows for flexible programming patterns, but presents challenges for type systems. In particular, a simple treatment of method extraction would require methods to be contravariant in the receiver type, making overriding all-but-impossible. We present a detailed investigation of this problem, as well as an implemented and evaluated solution.
Method extraction is a feature of many dynamically-typed and gradually-typed languages, ranging from Python and PHP to Flow and TypeScript. In these languages, the underlying representation of objects as records of procedures can be accessed, and the procedures that implement methods can be reified as functions that can be called independently. In many of these languages, the programmer can then explicitly specify the this value to be used when the method implementation is called.
Unfortunately, as we show, existing gradual type systems such as TypeScript and Flow are unsound in the presence of method extraction. The problem for typing any such system is that the flexibility it allows must be tamed by requiring a connection between the object the method was extracted from, and the function value that is later called.
In Racket, where a method extraction-like facility, dubbed “structure type properties”, is fundamental to classes, generic methods, and other APIs, these same challenges arise, and must be solved to support this feature in Typed Racket. We show how to combine two existing type system features—existential types and occurrence typing—to produce a sound approach to typing method extraction.
We formalize our design, extending an existing formal model of the Typed Racket type system, and prove that our extension is sound. Our design is also implemented in the released version of Racket, and is compatible with all existing Typed Racket packages, many of which already used a previous version of this feature.
@Article{Programming Journal, Volume22p6,
author = {Yuquan Fu and Sam Tobin-Hochstadt},
title = {Type Checking Extracted Methods},
journal = {},
volume = {},
number = {},
articleno = {6},
numpages = {43},
doi = {10.22152/programming-journal.org/2022/6/6},
year = {2022},
}
Publisher's Version
Programming Journal, Volume '22: "Continuation-Passing Style, ..."
Continuation-Passing Style, Defunctionalization, Accumulations, and Associativity
Jeremy Gibbons
(University of Oxford, UK)
Context: Reynolds showed us how to use continuation-passing style and defunctionalization to transform a recursive interpreter for a language into an abstract machine for programs in that language. The same techniques explain other programming tricks, including zippers and accumulating parameters. Inquiry: Buried within all those applications there is usually a hidden appeal to the algebraic property of associativity. Approach: Our purpose in this paper is to entice associativity out of the shadows and into the limelight. Knowledge: We revisit some well-known applications (factorial, fast reverse, tree flattening, and a compiler for a simple expression language) to spotlight their dependence on associativity. Grounding: We replay developments of these programs through a series of program transformations and data refinements, justified by equational reasoning. Importance: Understanding the crucial role played by associativity clarifies when continuation-passing style and defunctionalization can help and when they cannot, and may prompt other applications of these techniques.
@Article{Programming Journal, Volume22p7,
author = {Jeremy Gibbons},
title = {Continuation-Passing Style, Defunctionalization, Accumulations, and Associativity},
journal = {},
volume = {},
number = {},
articleno = {7},
numpages = {28},
doi = {10.22152/programming-journal.org/2022/6/7},
year = {2022},
}
Publisher's Version
Programming Journal, Volume '22: "Types for Tables: A Language ..."
Types for Tables: A Language Design Benchmark
Kuang-Chen Lu, Ben Greenman, and Shriram Krishnamurthi
(Brown University, USA)
Context Tables are ubiquitous formats for data. Therefore, techniques for writing correct programs over tables, and debugging incorrect ones, are vital. Our specific focus in this paper is on rich types that articulate the properties of tabular operations. We wish to study both their expressive power and diagnostic quality. Inquiry There is no "standard library" of table operations. As a result, every paper (and project) is free to use its own (sub)set of operations. This makes artifacts very difficult to compare, and it can be hard to tell whether omitted operations were left out by oversight or because they cannot actually be expressed. Furthermore, virtually no papers discuss the quality of type error feedback. Approach We combed through several existing languages and libraries to create a "standard library" of table operations. Each entry is accompanied by a detailed specification of its "type," expressed independent of (and hence not constrained by) any type language. We also studied and categorized a corpus of (student) program edits that resulted in table-related errors. We used this to generate a suite of erroneous programs. Finally, we adapted the concept of a datasheet to facilitate comparisons of different implementations. Knowledge Our benchmark creates a common ground to frame work in this area. Language designers who claim to support typed programming over tables have a clear suite against which to demonstrate their system's expressive power. Our family of errors also gives them a chance to demonstrate the quality of feedback. Researchers who improve one aspect---especially error reporting---without changing the other can demonstrate their improvement, as can those who engage in trade-offs between the two. The net result should be much better science in both expressiveness and diagnostics. We also introduce a datasheet format for presenting this knowledge in a methodical way. Grounding We have generated our benchmark from real languages, libraries, and programs, as well as personal experience conducting and teaching data science. We have drawn on experience in engineering and, more recently, in data science to generate the datasheet. Importance Claims about type support for tabular programming are hard to evaluate. However, tabular programming is ubiquitous, and the expressive power of type systems keeps growing. Our benchmark and datasheet can help lead to more orderly science. It also benefits programmers trying to choose a language.
@Article{Programming Journal, Volume22p8,
author = {Kuang-Chen Lu and Ben Greenman and Shriram Krishnamurthi},
title = {Types for Tables: A Language Design Benchmark},
journal = {},
volume = {},
number = {},
articleno = {8},
numpages = {30},
doi = {10.22152/programming-journal.org/2022/6/8},
year = {2022},
}
Publisher's Version
Info
Programming Journal, Volume '22: "A Transient Semantics for ..."
A Transient Semantics for Typed Racket
Ben Greenman, Lukas Lazarek, Christos Dimoulas, and Matthias Felleisen
(Brown University, USA; Northeastern University, USA; Northwestern University, USA)
Mixed-typed languages enable programmers to link typed and untyped components in various ways. Some offer rich type systems to facilitate the smooth migration of untyped code to the typed world; others merely provide a convenient form of type Dynamic together with a conventional structural type system. Orthogonal to this dimension, Natural systems ensure the integrity of types with a sophisticated contract system, while Transient systems insert simple first-order checks at strategic places within typed code. Furthermore, each method of ensuring type integrity comes with its own blame-assignment strategy.
Typed Racket has a rich migratory type system and enforces the types with a Natural semantics. Reticulated Python has a simple structural type system extended with Dynamic and enforces types with a Transient semantics. While Typed Racket satisfies the most stringent gradual-type soundness properties at a significant performance cost, Reticulated Python seems to limit the performance penalty to a tolerable degree and is nevertheless type sound. This comparison raises the question of whether Transient checking is applicable to and beneficial for a rich migratory type system.
This paper reports on the surprising difficulties of adapting the Transient semantics of Reticulated Python to the rich migratory type system of Typed Racket. The resulting implementation, Shallow Typed Racket, is faster than the standard Deep Typed Racket but only when the Transient blame assignment strategy is disabled. For language designers, this report provides valuable hints on how to equip an existing compiler to support a Transient semantics. For theoreticians, the negative experience with Transient blame calls for a thorough investigation of this strategy.
@Article{Programming Journal, Volume22p9,
author = {Ben Greenman and Lukas Lazarek and Christos Dimoulas and Matthias Felleisen},
title = {A Transient Semantics for Typed Racket},
journal = {},
volume = {},
number = {},
articleno = {9},
numpages = {26},
doi = {10.22152/programming-journal.org/2022/6/9},
year = {2022},
}
Publisher's Version
Programming Journal, Volume '22: "Types for Tables: A Language ..."
Types for Tables: A Language Design Benchmark
Kuang-Chen Lu, Ben Greenman, and Shriram Krishnamurthi
(Brown University, USA)
Context Tables are ubiquitous formats for data. Therefore, techniques for writing correct programs over tables, and debugging incorrect ones, are vital. Our specific focus in this paper is on rich types that articulate the properties of tabular operations. We wish to study both their expressive power and diagnostic quality. Inquiry There is no "standard library" of table operations. As a result, every paper (and project) is free to use its own (sub)set of operations. This makes artifacts very difficult to compare, and it can be hard to tell whether omitted operations were left out by oversight or because they cannot actually be expressed. Furthermore, virtually no papers discuss the quality of type error feedback. Approach We combed through several existing languages and libraries to create a "standard library" of table operations. Each entry is accompanied by a detailed specification of its "type," expressed independent of (and hence not constrained by) any type language. We also studied and categorized a corpus of (student) program edits that resulted in table-related errors. We used this to generate a suite of erroneous programs. Finally, we adapted the concept of a datasheet to facilitate comparisons of different implementations. Knowledge Our benchmark creates a common ground to frame work in this area. Language designers who claim to support typed programming over tables have a clear suite against which to demonstrate their system's expressive power. Our family of errors also gives them a chance to demonstrate the quality of feedback. Researchers who improve one aspect---especially error reporting---without changing the other can demonstrate their improvement, as can those who engage in trade-offs between the two. The net result should be much better science in both expressiveness and diagnostics. We also introduce a datasheet format for presenting this knowledge in a methodical way. Grounding We have generated our benchmark from real languages, libraries, and programs, as well as personal experience conducting and teaching data science. We have drawn on experience in engineering and, more recently, in data science to generate the datasheet. Importance Claims about type support for tabular programming are hard to evaluate. However, tabular programming is ubiquitous, and the expressive power of type systems keeps growing. Our benchmark and datasheet can help lead to more orderly science. It also benefits programmers trying to choose a language.
@Article{Programming Journal, Volume22p8,
author = {Kuang-Chen Lu and Ben Greenman and Shriram Krishnamurthi},
title = {Types for Tables: A Language Design Benchmark},
journal = {},
volume = {},
number = {},
articleno = {8},
numpages = {30},
doi = {10.22152/programming-journal.org/2022/6/8},
year = {2022},
}
Publisher's Version
Info
Programming Journal, Volume '22: "Automated, Targeted Testing ..."
Automated, Targeted Testing of Property-Based Testing Predicates
Tim Nelson, Elijah Rivera, Sam Soucie, Thomas Del Vecchio, John Wrenn, and Shriram Krishnamurthi
(Brown University, USA; Massachusetts Institute of Technology, USA; Indiana University, USA)
Context This work is based on property-based testing (PBT). PBT is an increasingly important form of software testing. Furthermore, it serves as a concrete gateway into the abstract area of formal methods. Specifically, we focus on students learning PBT methods.
Inquiry How well do students do at PBT? Our goal is to assess the quality of the predicates they write as part of PBT. Prior work introduced the idea of decomposing the predicate’s property into a conjunction of independent subproperties. Testing the predicate against each subproperty gives a “semantic” understanding of their performance.
Approach The notion of independence of subproperties both seems intuitive and was an important condition in prior work. First, we show that this condition is overly restrictive and might hide valuable information: it both undercounts errors and makes it hard to capture misconceptions. Second, we introduce two forms of automation, one based on PBT tools and the other on SAT-solving, to enable testing of student predicates. Third, we compare the output of these automated tools against manually-constructed tests. Fourth, we also measure the performance of those tools. Finally, we re-assess student performance reported in prior work.
Knowledge We show the difficulty caused by the independent subproperty requirement. We provide insight into how to use automation effectively to assess PBT predicates. In particular, we discuss the steps we had to take to beat human performance. We also provide insight into how to make the automation work efficiently. Finally, we present a much richer account than prior work of how students did.
Grounding Our methods are grounded in mathematical logic. We also make use of well-understood principles of test generation from more formal specifications. This combination ensures the soundness of our work. We use standard methods to measure performance.
Importance As both educators and programmers, we believe PBT is a valuable tool for students to learn, and its importance will only grow as more developers appreciate its value. Effective teaching requires a clear understanding of student knowledge and progress. Our methods enable a rich and automated analysis of student performance on PBT that yields insight into their understanding and can capture misconceptions. We therefore expect these results to be valuable to educators.
@Article{Programming Journal, Volume22p10,
author = {Tim Nelson and Elijah Rivera and Sam Soucie and Thomas Del Vecchio and John Wrenn and Shriram Krishnamurthi},
title = {Automated, Targeted Testing of Property-Based Testing Predicates},
journal = {},
volume = {},
number = {},
articleno = {10},
numpages = {29},
doi = {10.22152/programming-journal.org/2022/6/10},
year = {2022},
}
Publisher's Version
Info
Programming Journal, Volume '22: "A Transient Semantics for ..."
A Transient Semantics for Typed Racket
Ben Greenman, Lukas Lazarek, Christos Dimoulas, and Matthias Felleisen
(Brown University, USA; Northeastern University, USA; Northwestern University, USA)
Mixed-typed languages enable programmers to link typed and untyped components in various ways. Some offer rich type systems to facilitate the smooth migration of untyped code to the typed world; others merely provide a convenient form of type Dynamic together with a conventional structural type system. Orthogonal to this dimension, Natural systems ensure the integrity of types with a sophisticated contract system, while Transient systems insert simple first-order checks at strategic places within typed code. Furthermore, each method of ensuring type integrity comes with its own blame-assignment strategy.
Typed Racket has a rich migratory type system and enforces the types with a Natural semantics. Reticulated Python has a simple structural type system extended with Dynamic and enforces types with a Transient semantics. While Typed Racket satisfies the most stringent gradual-type soundness properties at a significant performance cost, Reticulated Python seems to limit the performance penalty to a tolerable degree and is nevertheless type sound. This comparison raises the question of whether Transient checking is applicable to and beneficial for a rich migratory type system.
This paper reports on the surprising difficulties of adapting the Transient semantics of Reticulated Python to the rich migratory type system of Typed Racket. The resulting implementation, Shallow Typed Racket, is faster than the standard Deep Typed Racket but only when the Transient blame assignment strategy is disabled. For language designers, this report provides valuable hints on how to equip an existing compiler to support a Transient semantics. For theoreticians, the negative experience with Transient blame calls for a thorough investigation of this strategy.
@Article{Programming Journal, Volume22p9,
author = {Ben Greenman and Lukas Lazarek and Christos Dimoulas and Matthias Felleisen},
title = {A Transient Semantics for Typed Racket},
journal = {},
volume = {},
number = {},
articleno = {9},
numpages = {26},
doi = {10.22152/programming-journal.org/2022/6/9},
year = {2022},
}
Publisher's Version
Programming Journal, Volume '22: "Types for Tables: A Language ..."
Types for Tables: A Language Design Benchmark
Kuang-Chen Lu, Ben Greenman, and Shriram Krishnamurthi
(Brown University, USA)
Context Tables are ubiquitous formats for data. Therefore, techniques for writing correct programs over tables, and debugging incorrect ones, are vital. Our specific focus in this paper is on rich types that articulate the properties of tabular operations. We wish to study both their expressive power and diagnostic quality. Inquiry There is no "standard library" of table operations. As a result, every paper (and project) is free to use its own (sub)set of operations. This makes artifacts very difficult to compare, and it can be hard to tell whether omitted operations were left out by oversight or because they cannot actually be expressed. Furthermore, virtually no papers discuss the quality of type error feedback. Approach We combed through several existing languages and libraries to create a "standard library" of table operations. Each entry is accompanied by a detailed specification of its "type," expressed independent of (and hence not constrained by) any type language. We also studied and categorized a corpus of (student) program edits that resulted in table-related errors. We used this to generate a suite of erroneous programs. Finally, we adapted the concept of a datasheet to facilitate comparisons of different implementations. Knowledge Our benchmark creates a common ground to frame work in this area. Language designers who claim to support typed programming over tables have a clear suite against which to demonstrate their system's expressive power. Our family of errors also gives them a chance to demonstrate the quality of feedback. Researchers who improve one aspect---especially error reporting---without changing the other can demonstrate their improvement, as can those who engage in trade-offs between the two. The net result should be much better science in both expressiveness and diagnostics. We also introduce a datasheet format for presenting this knowledge in a methodical way. Grounding We have generated our benchmark from real languages, libraries, and programs, as well as personal experience conducting and teaching data science. We have drawn on experience in engineering and, more recently, in data science to generate the datasheet. Importance Claims about type support for tabular programming are hard to evaluate. However, tabular programming is ubiquitous, and the expressive power of type systems keeps growing. Our benchmark and datasheet can help lead to more orderly science. It also benefits programmers trying to choose a language.
@Article{Programming Journal, Volume22p8,
author = {Kuang-Chen Lu and Ben Greenman and Shriram Krishnamurthi},
title = {Types for Tables: A Language Design Benchmark},
journal = {},
volume = {},
number = {},
articleno = {8},
numpages = {30},
doi = {10.22152/programming-journal.org/2022/6/8},
year = {2022},
}
Publisher's Version
Info
Programming Journal, Volume '22: "Automated, Targeted Testing ..."
Automated, Targeted Testing of Property-Based Testing Predicates
Tim Nelson, Elijah Rivera, Sam Soucie, Thomas Del Vecchio, John Wrenn, and Shriram Krishnamurthi
(Brown University, USA; Massachusetts Institute of Technology, USA; Indiana University, USA)
Context This work is based on property-based testing (PBT). PBT is an increasingly important form of software testing. Furthermore, it serves as a concrete gateway into the abstract area of formal methods. Specifically, we focus on students learning PBT methods.
Inquiry How well do students do at PBT? Our goal is to assess the quality of the predicates they write as part of PBT. Prior work introduced the idea of decomposing the predicate’s property into a conjunction of independent subproperties. Testing the predicate against each subproperty gives a “semantic” understanding of their performance.
Approach The notion of independence of subproperties both seems intuitive and was an important condition in prior work. First, we show that this condition is overly restrictive and might hide valuable information: it both undercounts errors and makes it hard to capture misconceptions. Second, we introduce two forms of automation, one based on PBT tools and the other on SAT-solving, to enable testing of student predicates. Third, we compare the output of these automated tools against manually-constructed tests. Fourth, we also measure the performance of those tools. Finally, we re-assess student performance reported in prior work.
Knowledge We show the difficulty caused by the independent subproperty requirement. We provide insight into how to use automation effectively to assess PBT predicates. In particular, we discuss the steps we had to take to beat human performance. We also provide insight into how to make the automation work efficiently. Finally, we present a much richer account than prior work of how students did.
Grounding Our methods are grounded in mathematical logic. We also make use of well-understood principles of test generation from more formal specifications. This combination ensures the soundness of our work. We use standard methods to measure performance.
Importance As both educators and programmers, we believe PBT is a valuable tool for students to learn, and its importance will only grow as more developers appreciate its value. Effective teaching requires a clear understanding of student knowledge and progress. Our methods enable a rich and automated analysis of student performance on PBT that yields insight into their understanding and can capture misconceptions. We therefore expect these results to be valuable to educators.
@Article{Programming Journal, Volume22p10,
author = {Tim Nelson and Elijah Rivera and Sam Soucie and Thomas Del Vecchio and John Wrenn and Shriram Krishnamurthi},
title = {Automated, Targeted Testing of Property-Based Testing Predicates},
journal = {},
volume = {},
number = {},
articleno = {10},
numpages = {29},
doi = {10.22152/programming-journal.org/2022/6/10},
year = {2022},
}
Publisher's Version
Info
Programming Journal, Volume '22: "Automated, Targeted Testing ..."
Automated, Targeted Testing of Property-Based Testing Predicates
Tim Nelson, Elijah Rivera, Sam Soucie, Thomas Del Vecchio, John Wrenn, and Shriram Krishnamurthi
(Brown University, USA; Massachusetts Institute of Technology, USA; Indiana University, USA)
Context This work is based on property-based testing (PBT). PBT is an increasingly important form of software testing. Furthermore, it serves as a concrete gateway into the abstract area of formal methods. Specifically, we focus on students learning PBT methods.
Inquiry How well do students do at PBT? Our goal is to assess the quality of the predicates they write as part of PBT. Prior work introduced the idea of decomposing the predicate’s property into a conjunction of independent subproperties. Testing the predicate against each subproperty gives a “semantic” understanding of their performance.
Approach The notion of independence of subproperties both seems intuitive and was an important condition in prior work. First, we show that this condition is overly restrictive and might hide valuable information: it both undercounts errors and makes it hard to capture misconceptions. Second, we introduce two forms of automation, one based on PBT tools and the other on SAT-solving, to enable testing of student predicates. Third, we compare the output of these automated tools against manually-constructed tests. Fourth, we also measure the performance of those tools. Finally, we re-assess student performance reported in prior work.
Knowledge We show the difficulty caused by the independent subproperty requirement. We provide insight into how to use automation effectively to assess PBT predicates. In particular, we discuss the steps we had to take to beat human performance. We also provide insight into how to make the automation work efficiently. Finally, we present a much richer account than prior work of how students did.
Grounding Our methods are grounded in mathematical logic. We also make use of well-understood principles of test generation from more formal specifications. This combination ensures the soundness of our work. We use standard methods to measure performance.
Importance As both educators and programmers, we believe PBT is a valuable tool for students to learn, and its importance will only grow as more developers appreciate its value. Effective teaching requires a clear understanding of student knowledge and progress. Our methods enable a rich and automated analysis of student performance on PBT that yields insight into their understanding and can capture misconceptions. We therefore expect these results to be valuable to educators.
@Article{Programming Journal, Volume22p10,
author = {Tim Nelson and Elijah Rivera and Sam Soucie and Thomas Del Vecchio and John Wrenn and Shriram Krishnamurthi},
title = {Automated, Targeted Testing of Property-Based Testing Predicates},
journal = {},
volume = {},
number = {},
articleno = {10},
numpages = {29},
doi = {10.22152/programming-journal.org/2022/6/10},
year = {2022},
}
Publisher's Version
Info
Programming Journal, Volume '22: "Automated, Targeted Testing ..."
Automated, Targeted Testing of Property-Based Testing Predicates
Tim Nelson, Elijah Rivera, Sam Soucie, Thomas Del Vecchio, John Wrenn, and Shriram Krishnamurthi
(Brown University, USA; Massachusetts Institute of Technology, USA; Indiana University, USA)
Context This work is based on property-based testing (PBT). PBT is an increasingly important form of software testing. Furthermore, it serves as a concrete gateway into the abstract area of formal methods. Specifically, we focus on students learning PBT methods.
Inquiry How well do students do at PBT? Our goal is to assess the quality of the predicates they write as part of PBT. Prior work introduced the idea of decomposing the predicate’s property into a conjunction of independent subproperties. Testing the predicate against each subproperty gives a “semantic” understanding of their performance.
Approach The notion of independence of subproperties both seems intuitive and was an important condition in prior work. First, we show that this condition is overly restrictive and might hide valuable information: it both undercounts errors and makes it hard to capture misconceptions. Second, we introduce two forms of automation, one based on PBT tools and the other on SAT-solving, to enable testing of student predicates. Third, we compare the output of these automated tools against manually-constructed tests. Fourth, we also measure the performance of those tools. Finally, we re-assess student performance reported in prior work.
Knowledge We show the difficulty caused by the independent subproperty requirement. We provide insight into how to use automation effectively to assess PBT predicates. In particular, we discuss the steps we had to take to beat human performance. We also provide insight into how to make the automation work efficiently. Finally, we present a much richer account than prior work of how students did.
Grounding Our methods are grounded in mathematical logic. We also make use of well-understood principles of test generation from more formal specifications. This combination ensures the soundness of our work. We use standard methods to measure performance.
Importance As both educators and programmers, we believe PBT is a valuable tool for students to learn, and its importance will only grow as more developers appreciate its value. Effective teaching requires a clear understanding of student knowledge and progress. Our methods enable a rich and automated analysis of student performance on PBT that yields insight into their understanding and can capture misconceptions. We therefore expect these results to be valuable to educators.
@Article{Programming Journal, Volume22p10,
author = {Tim Nelson and Elijah Rivera and Sam Soucie and Thomas Del Vecchio and John Wrenn and Shriram Krishnamurthi},
title = {Automated, Targeted Testing of Property-Based Testing Predicates},
journal = {},
volume = {},
number = {},
articleno = {10},
numpages = {29},
doi = {10.22152/programming-journal.org/2022/6/10},
year = {2022},
}
Publisher's Version
Info
Programming Journal, Volume '22: "Type Checking Extracted Methods ..."
Type Checking Extracted Methods
Yuquan Fu and Sam Tobin-Hochstadt
(Indiana University, USA)
Many object-oriented dynamic languages allow programmers to extract methods from objects and treat them as functions. This allows for flexible programming patterns, but presents challenges for type systems. In particular, a simple treatment of method extraction would require methods to be contravariant in the receiver type, making overriding all-but-impossible. We present a detailed investigation of this problem, as well as an implemented and evaluated solution.
Method extraction is a feature of many dynamically-typed and gradually-typed languages, ranging from Python and PHP to Flow and TypeScript. In these languages, the underlying representation of objects as records of procedures can be accessed, and the procedures that implement methods can be reified as functions that can be called independently. In many of these languages, the programmer can then explicitly specify the this value to be used when the method implementation is called.
Unfortunately, as we show, existing gradual type systems such as TypeScript and Flow are unsound in the presence of method extraction. The problem for typing any such system is that the flexibility it allows must be tamed by requiring a connection between the object the method was extracted from, and the function value that is later called.
In Racket, where a method extraction-like facility, dubbed “structure type properties”, is fundamental to classes, generic methods, and other APIs, these same challenges arise, and must be solved to support this feature in Typed Racket. We show how to combine two existing type system features—existential types and occurrence typing—to produce a sound approach to typing method extraction.
We formalize our design, extending an existing formal model of the Typed Racket type system, and prove that our extension is sound. Our design is also implemented in the released version of Racket, and is compatible with all existing Typed Racket packages, many of which already used a previous version of this feature.
@Article{Programming Journal, Volume22p6,
author = {Yuquan Fu and Sam Tobin-Hochstadt},
title = {Type Checking Extracted Methods},
journal = {},
volume = {},
number = {},
articleno = {6},
numpages = {43},
doi = {10.22152/programming-journal.org/2022/6/6},
year = {2022},
}
Publisher's Version
Programming Journal, Volume '22: "Automated, Targeted Testing ..."
Automated, Targeted Testing of Property-Based Testing Predicates
Tim Nelson, Elijah Rivera, Sam Soucie, Thomas Del Vecchio, John Wrenn, and Shriram Krishnamurthi
(Brown University, USA; Massachusetts Institute of Technology, USA; Indiana University, USA)
Context This work is based on property-based testing (PBT). PBT is an increasingly important form of software testing. Furthermore, it serves as a concrete gateway into the abstract area of formal methods. Specifically, we focus on students learning PBT methods.
Inquiry How well do students do at PBT? Our goal is to assess the quality of the predicates they write as part of PBT. Prior work introduced the idea of decomposing the predicate’s property into a conjunction of independent subproperties. Testing the predicate against each subproperty gives a “semantic” understanding of their performance.
Approach The notion of independence of subproperties both seems intuitive and was an important condition in prior work. First, we show that this condition is overly restrictive and might hide valuable information: it both undercounts errors and makes it hard to capture misconceptions. Second, we introduce two forms of automation, one based on PBT tools and the other on SAT-solving, to enable testing of student predicates. Third, we compare the output of these automated tools against manually-constructed tests. Fourth, we also measure the performance of those tools. Finally, we re-assess student performance reported in prior work.
Knowledge We show the difficulty caused by the independent subproperty requirement. We provide insight into how to use automation effectively to assess PBT predicates. In particular, we discuss the steps we had to take to beat human performance. We also provide insight into how to make the automation work efficiently. Finally, we present a much richer account than prior work of how students did.
Grounding Our methods are grounded in mathematical logic. We also make use of well-understood principles of test generation from more formal specifications. This combination ensures the soundness of our work. We use standard methods to measure performance.
Importance As both educators and programmers, we believe PBT is a valuable tool for students to learn, and its importance will only grow as more developers appreciate its value. Effective teaching requires a clear understanding of student knowledge and progress. Our methods enable a rich and automated analysis of student performance on PBT that yields insight into their understanding and can capture misconceptions. We therefore expect these results to be valuable to educators.
@Article{Programming Journal, Volume22p10,
author = {Tim Nelson and Elijah Rivera and Sam Soucie and Thomas Del Vecchio and John Wrenn and Shriram Krishnamurthi},
title = {Automated, Targeted Testing of Property-Based Testing Predicates},
journal = {},
volume = {},
number = {},
articleno = {10},
numpages = {29},
doi = {10.22152/programming-journal.org/2022/6/10},
year = {2022},
}
Publisher's Version
Info