Additional failing test cases in defects4j Lang project

[SZTankWang]

Context

Hi:
I am currently using defects4j and GZoltar together to evaluate the performance of fault localization. Therefore, I referred to the experiment setup in Evaluating and Improving Fault Localization Techniques.

Currently, I am facing the problem that additional test cases are failing when ran in isolation by GZoltar. I know that GZoltar is using JUnitCore to run those test cases. And I noticed that there have been previous issues on this problem, like D4J Issue 42. However, I haven't had a clear sense on how to solve the problem. Could you share with me how you tackled the problem in "Evaluating and Improving Fault Localization Techniques", since you were also using GZoltar.

Steps to Reproduce

Defects4j Lang; bug id 59.
Java version: 1.8.0_312
GZoltar version: 1.7.3-SNAPSHOT
JUnit Version: 3.8.1 (following the pom.xml in Lang59)

GZoltar arguments

I am following the original GZoltar command line file's arguments.

Expected behaviour

For Lang 59, there are these tests failing additionally:

org.apache.commons.lang.EntitiesPerformanceTest#testUnescapeArray has finished! Has it failed? true
org.apache.commons.lang.EntitiesPerformanceTest#testEscapeArray has finished! Has it failed? true
org.apache.commons.lang.EntitiesPerformanceTest#testLookupHash has finished! Has it failed? true
org.apache.commons.lang.EntitiesPerformanceTest#testLookupTree has finished! Has it failed? true
org.apache.commons.lang.EntitiesPerformanceTest#testLookupArray has finished! Has it failed? true

And the trace info are similar:

java.lang.NullPointerException
	at org.apache.commons.lang.EntitiesPerformanceTest.lookup(EntitiesPerformanceTest.java:216)
	at org.apache.commons.lang.EntitiesPerformanceTest.testLookupHash(EntitiesPerformanceTest.java:131)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at junit.framework.TestCase.runTest(TestCase.java:176)
	at junit.framework.TestCase.runBare(TestCase.java:141)
	at junit.framework.TestResult$1.protect(TestResult.java:122)
	at junit.framework.TestResult.runProtected(TestResult.java:142)
	at junit.framework.TestResult.run(TestResult.java:125)
	at junit.framework.TestCase.run(TestCase.java:129)
	at junit.framework.TestSuite.runTest(TestSuite.java:255)
	at junit.framework.TestSuite.run(TestSuite.java:250)
	at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:84)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:160)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:138)
	at com.gzoltar.internal.core.test.junit.JUnitTestTask.call(JUnitTestTask.java:67)
	at com.gzoltar.internal.core.test.junit.JUnitTestTask.call(JUnitTestTask.java:27)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.lang.Thread.run(Thread.java:748)

Environment (please complete the following information, if relevant):

Ubuntu 20.04.2

Additional comments

I checked the test file, and a lot of methods in it were commented out as "Defects4j flaky methods ".

I noticed that in Evaluating and Improving Fault Localization Techniques, there were logics that handle the case when GZoltar and D4J do not agree with the failing test cases. It would be great if you could share a bit on how you handle the situation.

I would really appreciate it if you could provide some insights. Thank you!

[jose]

Hi @SZTankWang,

Thanks for reporting this issue. As far I'm aware, only a single additional non-revealing test case (org.joda.time.TestPeriodType::testForFields4) is failing in Time-{18, 22, 24, and 27} bugs, as described in here.

You mentioned that you're using "GZoltar version: 1.7.3-SNAPSHOT", could you please try the latest version (either 1.7.3 or 1.7.4-SNAPSHOT) and let me know whether the set of failing tests includes non-expected failing tests?

--
Best,
Jose