Evaluation of Mutation Testing in a Nuclear Industry Case Study

Pedro Delgado-Pérez, Ibrahim Habli, Steve Gregory, Robert David Alexander, John Andrew Clark, Inmaculada Medina-Bulo

Research output: Contribution to journalArticlepeer-review


For software quality assurance, many safety-critical industries appeal to the use of dynamic testing and structural coverage criteria. However, there are reasons to doubt the adequacy of such practices. Mutation testing has been suggested as an alternative or complementary approach but its cost has traditionally hindered its adoption by industry, and there are limited studies applying it to real safety-critical code. This paper evaluates the effectiveness of state-of-the-art mutation testing on safety-critical code from within the UK nuclear industry, in terms of revealing flaws in test suites that already meet the structural coverage criteria recommended by relevant safety standards. It also assesses the practical feasibility of implementing such mutation testing in a real setting. We applied a conventional selective mutation approach to a C codebase supplied by a nuclear industry partner and measured the mutation score achieved by the existing test suite. We repeated the experiment using trivial compiler equivalence (TCE) to assess the benefit that it might provide. Using a conventional approach, it first appeared that the existing test suite only killed 82% of the mutants, but applying TCE revealed that it killed 92%. The difference was due to equivalent or duplicate mutants that TCE eliminated. We then added new tests to kill all the surviving mutants, increasing the test suite size by 18% in the process. In conclusion, mutation testing can potentially improve fault detection compared to structural- coverage-guided testing, and may be affordable in a nuclear industry context. The industry feedback on our results was positive, although further evidence is needed from application of mutation testing to software with known real faults.
Original languageEnglish
Article number8453820
Pages (from-to)1406-1419
Number of pages14
JournalIEEE Transactions on Reliability
Issue number4
Early online date3 Sept 2018
Publication statusPublished - Dec 2018

Bibliographical note

© IEEE, 2018. This is an author-produced version of the published paper. Uploaded in accordance with the publisher’s self-archiving policy. Further copying may not be permitted; contact the publisher for details


  • Mutation Testing
  • safety-critical embedded systems
  • safety-critical systems
  • Coverage criteria
  • mutation testing
  • verification and validation
  • nuclear industry

Cite this