Flakey Tests

[nsoft]

Our build is currently plagued by tests that occasionally fail when they run via github actions, but don't seem to fail when running locally. My strong suspicion is that the difference lies in the availability of processing cores and stalled threads are not reviving before timeouts in the tests. This is often resolvable by simply re-running the workflow, but that's annoying and leaves us looking like there are problems. This issue will collect and count the various failures. I have an idea for a solution that will first show up in the test I am writing now for SendToOpenSearchProcessor but for now I'm keeping track here in case the pattern sheds additional light.

This table will track counts, comments will document occurances

Test	Count
org.jesterj.ingest.indexing.JdbcScannerImplFTITest > testScanWithMemory FAILED	6
org.jesterj.ingest.indexing.HeuristicFatalFTITest > test3errorsMarkedDead	3
org.jesterj.ingest.scanners.SimpleFileScannerImplTest > testFilter FAILED	1

[nsoft]

 org.jesterj.ingest.indexing.JdbcScannerImplFTITest > testScanWithMemory FAILED
    java.lang.AssertionError: expected:<14> but was:<15>
        at org.junit.Assert.fail(Assert.java:89)
        at org.junit.Assert.failNotEquals(Assert.java:835)
        at org.junit.Assert.assertEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:633)
        at org.jesterj.ingest.indexing.JdbcScannerImplFTITest.testScanWithMemory(JdbcScannerImplFTITest.java:167)

https://github.com/nsoft/jesterj/actions/runs/16510600501/job/46691654505

[nsoft]

Previously on Jul 21, 2025, 8:31 PM EDT

org.jesterj.ingest.scanners.SimpleFileScannerImplTest > testFilter FAILED
    java.lang.AssertionError: expected:<43> but was:<44>
        at org.junit.Assert.fail(Assert.java:89)
        at org.junit.Assert.failNotEquals(Assert.java:835)
        at org.junit.Assert.assertEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:633)
        at org.jesterj.ingest.scanners.SimpleFileScannerImplTest.testFilter(SimpleFileScannerImplTest.java:87)

https://github.com/nsoft/jesterj/actions/runs/16431164743/job/46432714132

[nsoft]

Often re-runs succeed, but this one failed on 2 tests:

org.jesterj.ingest.indexing.HeuristicFatalFTITest > test3errorsMarkedDead FAILED
    java.lang.AssertionError: Wrong number DEAD!
    C_OFI:38
    C_OFD:0
    JO:38
    No Index Stat C*:{file_countStepOther=[file:///tmp/src/jesterj/code/ingest/src/test/resources/test-data/comedies/winterstale, file:///tmp/src/jesterj/code/ingest/src/test/resources/test-data/comedies/allswellthatendswell, file:///tmp/src/jesterj/code/ingest/src/test/resources/test-data/comedies/twogentlemenofverona, file:///tmp/src/jesterj/code/ingest/src/test/resources/test-data/histories/2kinghenryiv, file:///tmp/src/jesterj/code/ingest/src/test/resources/test-data/histories/kinghenryviii, file:///tmp/src/jesterj/code/ingest/src/test/resources/test-data/comedies/tamingoftheshrew]}
    Multiple Index Stat C*:{file_countStepOther=[]}
     expected:<6> but was:<0>
        at org.junit.Assert.fail(Assert.java:89)
        at org.junit.Assert.failNotEquals(Assert.java:835)
        at org.junit.Assert.assertEquals(Assert.java:647)
        at org.jesterj.ingest.indexing.HeuristicFatalFTITest.test3errorsMarkedDead(HeuristicFatalFTITest.java:123)

org.jesterj.ingest.indexing.JdbcScannerImplFTITest > testScanWithMemory FAILED
    java.lang.AssertionError: [jdbc:hsqldb:mem:shakespeare;ifexists=true/play/5
    jdbc:hsqldb:mem:shakespeare;ifexists=true/play/1
    jdbc:hsqldb:mem:shakespeare;ifexists=true/play/2
    jdbc:hsqldb:mem:shakespeare;ifexists=true/play/3
    jdbc:hsqldb:mem:shakespeare;ifexists=true/play/29
    jdbc:hsqldb:mem:shakespeare;ifexists=true/play/4
    jdbc:hsqldb:mem:shakespeare;ifexists=true/play/31
    jdbc:hsqldb:mem:shakespeare;ifexists=true/play/30
    jdbc:hsqldb:mem:shakespeare;ifexists=true/play/17
    jdbc:hsqldb:mem:shakespeare;ifexists=true/play/39
    jdbc:hsqldb:mem:shakespeare;ifexists=true/play/38
    jdbc:hsqldb:mem:shakespeare;ifexists=true/play/16
    jdbc:hsqldb:mem:shakespeare;ifexists=true/play/15
    jdbc:hsqldb:mem:shakespeare;ifexists=true/play/37
    jdbc:hsqldb:mem:shakespeare;ifexists=true/play/14] expected:<10> but was:<15>
        at org.junit.Assert.fail(Assert.java:89)
        at org.junit.Assert.failNotEquals(Assert.java:835)
        at org.junit.Assert.assertEquals(Assert.java:647)
        at org.jesterj.ingest.indexing.JdbcScannerImplFTITest.testScanWithMemory(JdbcScannerImplFTITest.java:151)

This last one is interesting because it's the first case of expecting less, but this could be a thread doing the judgement getting delayed...

https://github.com/nsoft/jesterj/actions/runs/16510600501/job/46742717612

Identical on https://github.com/nsoft/jesterj/actions/runs/16581979412/job/46899856596?pr=209

Again as a pair https://github.com/nsoft/jesterj/actions/runs/16659655422/job/47153446269#step:7:548

[nsoft]

org.jesterj.ingest.indexing.JdbcScannerImplFTITest > testScanWithMemory FAILED
    java.lang.AssertionError: [jdbc:hsqldb:mem:shakespeare;ifexists=true/play/31
    jdbc:hsqldb:mem:shakespeare;ifexists=true/play/5
    jdbc:hsqldb:mem:shakespeare;ifexists=true/play/30
    jdbc:hsqldb:mem:shakespeare;ifexists=true/play/1
    jdbc:hsqldb:mem:shakespeare;ifexists=true/play/2
    jdbc:hsqldb:mem:shakespeare;ifexists=true/play/3
    jdbc:hsqldb:mem:shakespeare;ifexists=true/play/29
    jdbc:hsqldb:mem:shakespeare;ifexists=true/play/4
    jdbc:hsqldb:mem:shakespeare;ifexists=true/play/17
    jdbc:hsqldb:mem:shakespeare;ifexists=true/play/39
    jdbc:hsqldb:mem:shakespeare;ifexists=true/play/16] expected:<10> but was:<11>
        at org.junit.Assert.fail(Assert.java:89)
        at org.junit.Assert.failNotEquals(Assert.java:835)
        at org.junit.Assert.assertEquals(Assert.java:647)
        at org.jesterj.ingest.indexing.JdbcScannerImplFTITest.testScanWithMemory(JdbcScannerImplFTITest.java:151)

https://github.com/nsoft/jesterj/actions/runs/16657945210/job/47147638439#step:7:87

https://github.com/nsoft/jesterj/actions/runs/16662867912/job/47163610193#step:7:93

[nsoft]

    java.lang.AssertionError: expected:<2> but was:<1>
        at org.junit.Assert.fail(Assert.java:89)
        at org.junit.Assert.failNotEquals(Assert.java:835)
        at org.junit.Assert.assertEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:633)
        at org.jesterj.ingest.indexing.NonLinearDiamondFTITest.testScanWithMemoryDiamondIntoThree(NonLinearDiamondFTITest.java:155)