Harden bulk persistence against database timeouts in resource-constrained environments

[JayVDZ]

Overview

During integration testing (Scenario 8, MediumLarge), a transient database timeout occurred when JIM attempted to persist ~5,273 staged objects in a single bulk save operation. The test had processed all objects successfully with 0 errors — the failure was purely an I/O timeout during the final flush to PostgreSQL.

This was caused by host resource pressure (devcontainer under load), not a code bug. However, two hardening measures could make JIM more resilient in resource-constrained environments.

Proposed Improvements

1. Increase Npgsql command timeout for bulk operations

The default Npgsql command timeout (typically 30 seconds) may be insufficient for large batch saves. The import save operation could set a longer CommandTimeout on the DbContext during bulk persistence phases.

2. Batch the final save into smaller chunks

Instead of saving all objects in a single SaveChangesAsync() call, the import could flush in smaller batches (e.g., every 500–1,000 objects). This would reduce peak database load and make the operation more resilient to resource pressure.

Context

• Error: Exception while reading from stream --> Timeout during reading attempt
• Phase: Source Full Import, "Saving changes" step
• Objects processed: 5,273 (all successful, 0 errors)
• Environment: Devcontainer under resource pressure (VSCode update + performance issues)
• The test passes on a healthy host — this is a resilience improvement, not a bug fix