By the same authors

From the same journal

Evaluation of Mutation Testing in a Nuclear Industry Case Study

Research output: Contribution to journalArticle

Full text download(s)

Author(s)

Department/unit(s)

Publication details

JournalIEEE Transactions on Reliability
DateAccepted/In press - 5 Aug 2018
DatePublished (current) - Aug 2018
Issue number4
Volume67
Number of pages15
Pages (from-to)1406-1419
Original languageEnglish

Abstract

For software quality assurance, many safety-critical industries appeal to the use of dynamic testing and structural coverage criteria. However, there are reasons to doubt the adequacy of such practices. Mutation testing has been suggested as an alternative or complementary approach but its cost has traditionally hindered its adoption by industry, and there are limited studies applying it to real safety-critical code. This paper evaluates the effectiveness of state-of-the-art mutation testing on safety-critical code from within the UK nuclear industry, in terms of revealing flaws in test suites that already meet the structural coverage criteria recommended by relevant safety standards. It also assesses the practical feasibility of implementing such mutation testing in a real setting. We applied a conventional selective mutation approach to a C codebase supplied by a nuclear industry partner and measured the mutation score achieved by the existing test suite. We repeated the experiment using trivial compiler equivalence (TCE) to assess the benefit that it might provide. Using a conventional approach, it first appeared that the existing test suite only killed 82% of the mutants, but applying TCE revealed that it killed 92%. The difference was due to equivalent or duplicate mutants that TCE eliminated. We then added new tests to kill all the surviving mutants, increasing the test suite size by 18% in the process. In conclusion, mutation testing can potentially improve fault detection compared to structural- coverage-guided testing, and may be affordable in a nuclear industry context. The industry feedback on our results was positive, although further evidence is needed from application of mutation testing to software with known real faults.

Bibliographical note

© IEEE, 2018. This is an author-produced version of the published paper. Uploaded in accordance with the publisher’s self-archiving policy. Further copying may not be permitted; contact the publisher for details

    Research areas

  • Mutation Testing, safety-critical embedded systems, COVERAGE, VERIFICATION, NUCLEAR

Discover related content

Find related publications, people, projects, datasets and more using interactive charts.

View graph of relations