Skip to content

Add IP2M METRR data import utility#159

Open
cahaseler wants to merge 2 commits intomainfrom
import-utility-from-main
Open

Add IP2M METRR data import utility#159
cahaseler wants to merge 2 commits intomainfrom
import-utility-from-main

Conversation

@cahaseler
Copy link
Owner

Summary

This PR adds a utility to import legacy assessment data from the IP2M METRR system into EMPACT. The utility reads Excel exports from the legacy system and creates all necessary records in EMPACT while preserving data integrity.

Features

  • Excel Import: Reads standardized Excel exports with assessment and response data
  • Dry Run Mode: Preview what will be imported before making changes
  • Transaction Safety: All-or-nothing import using database transactions
  • Data Mapping: Handles status mapping, level mapping, and HTML cleaning
  • User Creation: Creates placeholder users for legacy user IDs
  • Comprehensive Documentation: Detailed Excel format specification and usage guide

Usage

# Preview import
yarn import:ip2m --file="data.xlsx" --dryRun

# Actual import
yarn import:ip2m --file="data.xlsx"

Import Results

Successfully imported test data:

  • 1 assessment collection
  • 1 assessment with 2 parts (Environment & Maturity)
  • 83 assessment attributes
  • 16 users (imported_ip2m_user_XXX@doe.gov)
  • 1 user group
  • 466 responses

Implementation Notes

  • Skips score summaries due to database schema mismatch
  • Uses placeholder users since these aren't real system users
  • Validates all attributes exist before import
  • Maps legacy level numbers to EMPACT level IDs
  • Updated to handle new database schema with assessmentUserGroupId

See web/scripts/README.md for complete documentation including Excel format specification.

- Created import script to migrate legacy assessment data from IP2M METRR Excel exports
- Supports dry-run mode for preview before actual import
- Imports assessments, users, responses with proper relationships
- Uses database transactions for data integrity
- Comprehensive documentation with Excel format specification
- Added yarn import:ip2m command for easy usage
- Handles new database schema with assessmentUserGroupId in responses
@cahaseler cahaseler requested a review from Copilot July 1, 2025 03:12
Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR introduces a new utility to import legacy IP2M METRR assessment data into EMPACT, along with configuration and documentation to support it.

  • Adds a TypeScript import script (import-ip2m-data-v2.ts) that reads Excel, validates, and imports data in a transaction
  • Introduces a dedicated tsconfig for scripts and updates package.json to run the import via yarn import:ip2m
  • Provides detailed documentation (web/scripts/README.md) and a general guidance file (CLAUDE.md)

Reviewed Changes

Copilot reviewed 5 out of 6 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
web/scripts/tsconfig.json New tsconfig for the scripts folder
web/scripts/import-ip2m-data-v2.ts Import utility implementation with validation, mapping, and transactional import
web/scripts/README.md Usage guide and Excel format specification
web/package.json Added import:ip2m script and xlsx dependency, fixed trailing commas
CLAUDE.md High-level guidance for Claude AI code interactions
Comments suppressed due to low confidence (2)

web/scripts/import-ip2m-data-v2.ts:58

  • This new import utility contains complex logic but has no automated tests. Consider adding unit or integration tests to cover data mapping, transaction flows, and error handling.
async function main() {

CLAUDE.md:1

  • [nitpick] This file appears unrelated to the IP2M import utility and may introduce noise. Consider moving it to a separate PR or clearly marking its scope.
# CLAUDE.md

Repository owner deleted a comment from Copilot AI Jul 1, 2025
Repository owner deleted a comment from Copilot AI Jul 1, 2025
- Replace custom HTML tag removal with sanitize-html library
- Fixes incomplete multi-character sanitization vulnerability
- Move sanitize-html to devDependencies as it's only used in scripts
@sonarqubecloud
Copy link

sonarqubecloud bot commented Jul 1, 2025

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants