Search-based Crash Reproduction using Behavioral Model Seeding (journal first)

Abstract

Search-based crash reproduction approaches assist developers during debugging by generating a test case, which reproduces a crash given its stack trace. One of the fundamental steps of this approach is creating objects needed to trigger the crash. One way to overcome this limitation is seeding: using information about the application during the search process. With seeding, existing classes usages participate in the search process to produce realistic sequences of method calls, which create the required objects. In this study, we introduce behavioural model seeding: a new seeding method that learns class usages from both the system under test and existing test cases. We synthesize learned usages in a behavioural model (state machine). Then, this model serves to guide the evolutionary process. To assess behavioural model seeding, we evaluate it against test seeding (the state-of-the-art technique for seeding realistic objects used in unit test generation) and no seeding (without seeding any class usage). For this evaluation, we use a benchmark of 122 hard-to-reproduce crashes stemming from six open-source projects. Our results indicate that model seeding outperforms other seeding approaches in all aspects: crash reproduction effectiveness, efficiency, and search process initialization rate. Model seeding increases the number of reproduced crashes by 7% and 6% compared to no seeding and test seeding, respectively. We manually investigate the improvements and outline three factors: dissimilarity between call sequences when sampling them from behaviour models, learning behavioural models from multiple information sources, and prioritizing classes to use for seeding. We make the following contributions: (i) we provide an evaluation of test seeding techniques applied to search-based crash reproduction, (ii) we design a novel behavioural model seeding strategy applied to search-based crash reproduction, (iii) we offer an open-source implementation of test seeding and model seeding strategies in the Botsing framework (https://github.com/STAMP-project/botsing), and (iv) we further discuss our model-seeding improvements in our replication package (https://doi.org/10.5281/zenodo.3673916). Our article is available open access (https://doi.org/10.1002/stvr.1733).

Date
Apr 15, 2021 —
Location
Online
Xavier Devroey
Xavier Devroey
Postdoctoral Researcher

My research interests include search-based and model-based software testing, test suite augmentation, DevOps, and variability-intensive systems engineering.

Related