Test: Concurrent Search Operations During Index Rebuilding

[nickna]

Description

Test system behavior when search operations are performed while background index rebuilding is in progress, ensuring no deadlocks or inconsistent results.

Phase

Phase 1: Chaos Engineering and Failure Injection

Epic

Related to #202

Acceptance Criteria

• Create tests with concurrent search operations during index builds
• Verify no deadlocks occur between search and indexing threads
• Ensure search results remain consistent and accurate
• Test with different search algorithms (Linear, KD-Tree, HNSW)
• Validate performance doesn't degrade severely during rebuilding

Concurrency Scenarios

1. Heavy search load during HNSW rebuild - Multiple concurrent searches
2. Search algorithm switching during build - Change algorithms mid-operation
3. Database modifications during index build - Add/remove vectors while indexing
4. Multiple index rebuilds triggered simultaneously - Race condition testing
5. Cancellation propagation - Cancel indexing while searches are active

Test Structure

[Test]
[Category("Chaos")]
public async Task ConcurrentSearches_DuringIndexRebuild_NoDeadlocks()
{
    // Arrange
    var database = new VectorDatabase();
    PopulateDatabase(database, vectorCount: 10000);
    
    var searchTasks = new List<Task<IList<Vector>>>();
    var indexBuildTask = Task.Run(() => database.RebuildSearchIndexAsync(SearchAlgorithm.HNSW));
    
    // Act: Start multiple concurrent searches
    for (int i = 0; i < 10; i++)
    {
        var query = CreateRandomQuery();
        searchTasks.Add(Task.Run(() => database.Search(query, 5, SearchAlgorithm.KDTree)));
    }
    
    // Wait for all operations
    await Task.WhenAll(searchTasks.Append(indexBuildTask));
    
    // Assert: All searches completed successfully
    foreach (var task in searchTasks)
    {
        Assert.That(task.Result.Count, Is.LessThanOrEqualTo(5));
        Assert.That(task.Status, Is.EqualTo(TaskStatus.RanToCompletion));
    }
}

Performance Monitoring

• Track search response times during indexing
• Monitor memory usage during concurrent operations
• Detect thread starvation or blocking