On the Predictability of Random Tests for Object-Oriented Software

Ilinca Ciupa; Alexander Pretschner; Andreas Leitner; Manuel Oriol; Bertrand Meyer

doi:10.1109/ICST.2008.20

On the Predictability of Random Tests for Object-Oriented Software

Ilinca Ciupa, Alexander Pretschner, Andreas Leitner, Manuel Oriol, Bertrand Meyer

Computer Science

Research output: Contribution to conference › Paper › peer-review

Abstract

Intuition suggests that random testing of object-oriented programs should exhibit a significant difference in the number of faults detected by two different runs of equal duration. As a consequence, random testing would be rather unpredictable. We evaluate the variance of the number of faults detected by random testing over time. We present the results of an empirical study that is based on 1215 hours of randomly testing 27 Eiffel classes, each with 30 seeds of the random number generator. Analyzing over 6 million failures triggered during the experiments, the study provides evidence that the relative number of faults detected by random testing over time is predictable but that different runs of the random test case generator detect different faults. The study also shows that random testing quickly finds faults: the first failure is likely to be triggered within 30 seconds.

Original language	Undefined/Unknown
Pages	72--81
DOIs	https://doi.org/10.1109/ICST.2008.20
Publication status	Published - 1 Apr 2008

Access to Document

10.1109/ICST.2008.20

http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=4539534

Cite this

@conference{8354dc14d3594cb5a34079c6b0927756,

title = "On the Predictability of Random Tests for Object-Oriented Software",

abstract = "Intuition suggests that random testing of object-oriented programs should exhibit a significant difference in the number of faults detected by two different runs of equal duration. As a consequence, random testing would be rather unpredictable. We evaluate the variance of the number of faults detected by random testing over time. We present the results of an empirical study that is based on 1215 hours of randomly testing 27 Eiffel classes, each with 30 seeds of the random number generator. Analyzing over 6 million failures triggered during the experiments, the study provides evidence that the relative number of faults detected by random testing over time is predictable but that different runs of the random test case generator detect different faults. The study also shows that random testing quickly finds faults: the first failure is likely to be triggered within 30 seconds.",

author = "Ilinca Ciupa and Alexander Pretschner and Andreas Leitner and Manuel Oriol and Bertrand Meyer",

year = "2008",

month = apr,

day = "1",

doi = "10.1109/ICST.2008.20",

language = "Undefined/Unknown",

pages = "72----81",

}

TY - CONF

T1 - On the Predictability of Random Tests for Object-Oriented Software

AU - Ciupa, Ilinca

AU - Pretschner, Alexander

AU - Leitner, Andreas

AU - Oriol, Manuel

AU - Meyer, Bertrand

PY - 2008/4/1

Y1 - 2008/4/1

N2 - Intuition suggests that random testing of object-oriented programs should exhibit a significant difference in the number of faults detected by two different runs of equal duration. As a consequence, random testing would be rather unpredictable. We evaluate the variance of the number of faults detected by random testing over time. We present the results of an empirical study that is based on 1215 hours of randomly testing 27 Eiffel classes, each with 30 seeds of the random number generator. Analyzing over 6 million failures triggered during the experiments, the study provides evidence that the relative number of faults detected by random testing over time is predictable but that different runs of the random test case generator detect different faults. The study also shows that random testing quickly finds faults: the first failure is likely to be triggered within 30 seconds.

AB - Intuition suggests that random testing of object-oriented programs should exhibit a significant difference in the number of faults detected by two different runs of equal duration. As a consequence, random testing would be rather unpredictable. We evaluate the variance of the number of faults detected by random testing over time. We present the results of an empirical study that is based on 1215 hours of randomly testing 27 Eiffel classes, each with 30 seeds of the random number generator. Analyzing over 6 million failures triggered during the experiments, the study provides evidence that the relative number of faults detected by random testing over time is predictable but that different runs of the random test case generator detect different faults. The study also shows that random testing quickly finds faults: the first failure is likely to be triggered within 30 seconds.

UR - http://www.scopus.com/inward/record.url?scp=50649100413&partnerID=8YFLogxK

U2 - 10.1109/ICST.2008.20

DO - 10.1109/ICST.2008.20

M3 - Paper

SP - 72

EP - 81

ER -

On the Predictability of Random Tests for Object-Oriented Software

Abstract

Access to Document

Other files and links

Cite this