CSV ingestion fixes #354
CSV ingestion fixes #354
Conversation
There was a problem hiding this comment.
Pull request overview
This pull request addresses CSV ingestion issues by removing the 6-track limit and adding comprehensive validation functionality to the CSV upload process. The changes enable teams to have more than 6 tracks and provide detailed error reporting before database insertion.
Changes:
- Removed
maxItems: 6constraint from teams' tracks array in database schema - Added two-step CSV validation and upload workflow with detailed error/warning reporting
- Implemented contact information extraction for easier follow-up on validation issues
Reviewed changes
Copilot reviewed 11 out of 11 changed files in this pull request and generated 17 comments.
Show a summary per file
| File | Description |
|---|---|
| migrations/create-teams.mjs | Removes maxItems: 6 from initial teams collection schema |
| migrations/20260105090000-remove-team-tracks-limit.mjs | New migration to remove track limit from existing databases |
| migrations/20250420035636-update-teams.mjs | Updates existing migration to remove track limit (concerning - modifies historical migration) |
| app/(pages)/admin/csv/page.tsx | Complete rewrite of CSV upload UI with validation, error reporting, and two-step workflow |
| app/(api)/_utils/csv-ingestion/csvAlgorithm.ts | Major enhancement with validation functions, contact extraction, and detailed issue reporting |
| app/(api)/_actions/logic/validateCSV.ts | New server action for CSV validation |
| app/(api)/_actions/logic/ingestTeams.ts | New server action for team ingestion |
| app/(api)/_actions/logic/checkTeamsPopulated.ts | New server action to check if teams exist |
| tests/csvValidation.test.ts | Tests for CSV validation behavior |
| tests/csvAlgorithm.test.ts | Tests for track matching and filtering |
| .prettierignore | Adds commented-out pattern for .mjs files (has syntax issue) |
Comments suppressed due to low confidence (1)
migrations/create-teams.mjs:42
- The maxItems constraint is being removed from the initial teams collection creation. While this is fine for new database setups, if this migration has already been run on production databases, this change won't affect them. The new migration 20260105090000-remove-team-tracks-limit.mjs is the correct approach for existing databases. However, for consistency in new setups, this change makes sense.
tracks: {
bsonType: 'array',
items: {
enum: tracks,
description: 'track must be one of the valid tracks',
},
description: 'tracks must be an array of strings',
},
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 14 out of 14 changed files in this pull request and generated 6 comments.
Comments suppressed due to low confidence (1)
migrations/20250420035636-update-teams.mjs:114
- The down() migration function is missing the 'reports' field that exists in the up() migration. This means if this migration is rolled back, the reports field will be removed from the schema validator, which could cause validation failures for existing documents that have reports. The down() function should include the same reports field definition as the up() function to maintain schema consistency during rollbacks.
tracks: {
bsonType: "array",
items: {
enum: tracks,
description: "track must be one of the valid tracks",
},
description: "tracks must be an array of strings",
},
active: {
bsonType: "bool",
description: "active must be a boolean",
},
},
additionalProperties: false,
},
},
});
};
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 14 out of 14 changed files in this pull request and generated 4 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| const validBody = results.filter((_, index) => { | ||
| // Find the original rowIndex for this result | ||
| for (const [rowIdx, outputIdx] of rowIndexToOutputIndex.entries()) { | ||
| if (outputIdx === index) { | ||
| return !errorRowIndexes.has(rowIdx); | ||
| } | ||
| } | ||
| // If somehow not found, include it (should not happen in practice) | ||
| return true; | ||
| }); |
There was a problem hiding this comment.
The rowIndexToOutputIndex mapping is built before teams are reordered (lines 441-442), but it's used after reordering (lines 483-492). When teams are reordered by putting "Best Hardware Hack" teams first (lines 454-467), the indices in the results array no longer match the outputIndex values stored in the map. This will cause the validBody filtering to incorrectly include or exclude teams when filtering out error rows.
To fix this, either:
- Build the rowIndexToOutputIndex map after reordering, or
- Store the rowIndex directly on each team object before reordering, or
- Use a Map keyed by teamNumber instead of array indices
|
Halo @haylietan / @michelleyeoh this is ready to merge now |
2 participants
Closes #349