Skip to content

feat(reindex): add /inventory-reindex-records/export API to export reindex record ranges to S3 and publish reindex.file-ready events#1318

Merged
psmagin merged 28 commits intomasterfrom
MODINVSTOR-1519
Mar 3, 2026
Merged

feat(reindex): add /inventory-reindex-records/export API to export reindex record ranges to S3 and publish reindex.file-ready events#1318
psmagin merged 28 commits intomasterfrom
MODINVSTOR-1519

Conversation

@psmagin
Copy link
Collaborator

@psmagin psmagin commented Feb 26, 2026

Purpose

Implement MODINVSTOR-1519 by adding export support for reindex record ranges so downstream reindex processing can consume NDJSON files from S3 and react to a file-ready Kafka event.

Approach

  • Added POST /inventory-reindex-records/export using the shared reindexRecordsRequest schema.
  • Implemented export orchestration and S3 upload flow (ReindexExportOrchestrator, ReindexS3ExportService) for instance, item, and holdings ranges.
  • Added reindex.file-ready event model/publisher and wired event emission after successful export.
  • Extended module configuration for dedicated reindex S3 settings (S3_REINDEX_*) and Kafka topic partitions (KAFKA_REINDEX_FILE_READY_TOPIC_NUM_PARTITIONS), and split MARC migration S3 settings to S3_MARC_MIGRATION_*.
  • Added/updated docs and tests for API behavior, orchestration, event publication, and export integration.

Changes Checklist

  • API Changes: Added POST /inventory-reindex-records/export; unified request schema/example names for reindex APIs (reindexRecordsRequest.json).
  • Database Schema Changes: No database schema changes.
  • Interface Version Changes: Bumped inventory-reindex-records to 1.1
  • Interface Dependencies: No interface dependency changes in this PR.
  • Permissions: Added inventory-storage.reindex-records.export.post and mapped endpoint permission in module descriptor.
  • Logging: Added/updated logging around reindex export processing.
  • Unit Testing: Added unit coverage for reindex export orchestrator, S3 export service, and file-ready event publisher.
  • Integration Testing: Added InventoryReindexRecordsExportIT and updated related integration tests.
  • Manual Testing: Tested on dev env.
  • NEWS: Added NEWS entry for /inventory-reindex-records/export.

Related Issues

Learning and Resources (if applicable)

N/A

Screenshots (if applicable)

N/A

@psmagin psmagin changed the title Modinvstor 1519 feat(reindex): add /inventory-reindex-records/export API to export reindex record ranges to S3 and publish reindex.file-ready events Feb 27, 2026
@psmagin psmagin marked this pull request as ready for review February 27, 2026 14:45
@psmagin psmagin requested a review from a team as a code owner February 27, 2026 14:45
@psmagin psmagin self-assigned this Feb 27, 2026
@viacheslavkol viacheslavkol self-requested a review March 2, 2026 14:19
@github-actions
Copy link

github-actions bot commented Mar 3, 2026

JSON Schema Changes (folio-org/mod-inventory-storage)

  • Base: ea6360f8f867a84147646edfc46a2e7391a8bca2
  • Head: 199d79e73c6de8c0e01aff1bf3ebe09b6713c144

Changed files

Status File (HEAD)
Renamed (100%) ramls/examples/reindex-records/reindexRecordsRequest.json
Renamed (082%) ramls/schemas/reindex-records/reindexRecordsRequest.json

Diffs (normalized first, fallback to raw if empty)

ramls/examples/reindex-records/reindexRecordsRequest.json

No structural changes (after normalization)

Showing raw textual diff (formatting/whitespace changes may appear)

ramls/schemas/reindex-records/reindexRecordsRequest.json

--- ramls/schemas/reindex-records/reindexRecordsRequest.json (base)
+++ ramls/schemas/reindex-records/reindexRecordsRequest.json (head)
@@ -2,7 +2,7 @@
   "$schema": "http://json-schema.org/draft-04/schema#",
   "additionalProperties": false,
   "description": "Inventory Storage records for reindex",
-  "javaType": "org.folio.rest.jaxrs.model.PublishReindexRecordsRequest",
+  "javaType": "org.folio.rest.jaxrs.model.ReindexRecordsRequest",
   "properties": {
     "id": {
       "$ref": "../common/uuid.json",
@@ -37,6 +37,11 @@
         "holdings"
       ],
       "type": "string"
+    },
+    "traceId": {
+      "$ref": "../common/uuid.json",
+      "description": "UUID for tracing the request, used for debugging and logging",
+      "type": "string"
     }
   },
   "required": [

@sonarqubecloud
Copy link

sonarqubecloud bot commented Mar 3, 2026

@psmagin psmagin merged commit 5022d15 into master Mar 3, 2026
19 checks passed
@psmagin psmagin deleted the MODINVSTOR-1519 branch March 3, 2026 13:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants