An Empirical Evaluation of Regular and Extreme Mutation Testing for Teaching Software Testing

Abstract

Teaching software testing can be challenging due to low student interest, high cognitive load, and lack of alignment with industry needs. Previous research has attempted to address these challenges by using mutation testing, which involves intentionally introducing faults into the code to measure the ability of a test suite to detect faults. Although this method has been proven effective in teaching software testing, it can sometimes be difficult for a novice to write a test to kill some mutants because they are too subtle and there are no hints. In contrast, extreme mutation testing involves more evident changes (e.g., removing a method body) that may be easier for novice testers to identify. This paper investigates extreme mutation testing as an alternative to teaching software testing by comparing it to regular mutation testing in an empirical evaluation with two undergraduate classes. Our results show that both can help teach software testing, with regular mutation testing slightly more effective, and both types of reports were considered clear by a similar number of students.