Regression test selection (RTS) aims to speed up regression testing by rerunning only tests that are affected by code changes. RTS can be performed using static or dynamic analysis techniques. Our recent study showed that static and dynamic RTS perform similarly for medium-sized Java projects. However, the results of that recent study also showed that static RTS can be unsafe, missing to select tests that dynamic RTS selects, and reflection was the only cause of unsafety observed among the evaluated projects.
In this paper, we investigate five techniques—three purely static techniques and two hybrid static-dynamic techniques—that aim to make static RTS safe with respect to reflection. We implemented these reflection-aware (RA) techniques by extending the reflection-unaware (RU) class-level static RTS technique in a tool called STARTS. To assess these RA techniques, we compared their end-to-end times with the RU technique, and with RetestAll which reruns all tests after every code change. We also compared safety and precision of the RA techniques with Ekstazi, a state-of-the-art dynamic RTS technique.
Our evaluation on 1173 versions of 24 open-source Java projects showed negative results. The RA techniques improve the safety of RU but at very high costs. The purely static techniques were safe in our experiments but decrease the precision of the RU technique, with end-to-end time at best 85.8% of RetestAll time, compared to 69.1% for RU. A hybrid static-dynamic technique provides better precision, is safer than RU and incurs lower end-to-end time—75.8% of RetestAll, but can still be unsafe in the presence of test-order dependencies. Our study highlights the challenges involved in making static RTS safe with respect to reflection.
This program is tentative and subject to change.
Wed 23 Oct
|14:00 - 14:22|
|14:22 - 14:45|
|14:45 - 15:07|
|15:07 - 15:30|