I think I identified another cause for the memory leaks observed in the tests.

The tests extending AbstractApiIntegrationTest use a slightly unusual pattern for setting up a test cluster:

security/src/integrationTest/java/org/opensearch/security/api/AbstractApiIntegrationTest.java

Lines 123 to 143 in d09ce78

    
           @Before 
        
           public void startCluster() throws IOException { 
        
               if (this.getClass().equals(testClass)) { 
        
                   return; 
        
               } 
        
               configurationFolder = ConfigurationFiles.createConfigurationDirectory(); 
        
               extendConfiguration(); 
        
               final var clusterManager = randomFrom(List.of(ClusterManager.THREE_CLUSTER_MANAGERS, ClusterManager.SINGLENODE)); 
        
               final var localClusterBuilder = new LocalCluster.Builder().clusterManager(clusterManager) 
        
                   .nodeSettings(getClusterSettings()) 
        
                   .defaultConfigurationInitDirectory(configurationFolder.toString()) 
        
                   .loadConfigurationIntoIndex(false); 
        
               localCluster = localClusterBuilder.build(); 
        
               localCluster.before(); 
        
               try (TestRestClient client = localCluster.getRestClient(ADMIN_USER_NAME, DEFAULT_PASSWORD)) { 
        
                   Awaitility.await() 
        
                       .alias("Load default configuration") 
        
                       .until(() -> client.securityHealth().getTextFromJsonBody("/status"), equalTo("UP")); 
        
               } 
        
               testClass = this.getClass(); 
        
           }

The cluster is stopped again here:

security/src/integrationTest/java/org/opensearch/security/api/AbstractApiIntegrationTest.java

Lines 265 to 269 in d09ce78

    
           @AfterClass 
        
           public static void stopCluster() throws IOException { 
        
               if (localCluster != null) localCluster.close(); 
        
               FileUtils.deleteDirectory(configurationFolder.toFile()); 
        
           }

This pattern is actually not safe, gradle runs the tests to a certain extent in parallel. This causes an existing active cluster instance getting overwritten in line 135. Thus, this cluster instance is never torn down and will live until the whole int test suite terminates.

	@Before
	public void startCluster() throws IOException {
	if (this.getClass().equals(testClass)) {
	return;
	}
	configurationFolder = ConfigurationFiles.createConfigurationDirectory();
	extendConfiguration();
	final var clusterManager = randomFrom(List.of(ClusterManager.THREE_CLUSTER_MANAGERS, ClusterManager.SINGLENODE));
	final var localClusterBuilder = new LocalCluster.Builder().clusterManager(clusterManager)
	.nodeSettings(getClusterSettings())
	.defaultConfigurationInitDirectory(configurationFolder.toString())
	.loadConfigurationIntoIndex(false);
	localCluster = localClusterBuilder.build();
	localCluster.before();
	try (TestRestClient client = localCluster.getRestClient(ADMIN_USER_NAME, DEFAULT_PASSWORD)) {
	Awaitility.await()
	.alias("Load default configuration")
	.until(() -> client.securityHealth().getTextFromJsonBody("/status"), equalTo("UP"));
	}
	testClass = this.getClass();
	}

	@AfterClass
	public static void stopCluster() throws IOException {
	if (localCluster != null) localCluster.close();
	FileUtils.deleteDirectory(configurationFolder.toFile());
	}

[Tests] Integration test issues #5915

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions