mirrulations-csv

A command line tool to convert the comment JSON files of a regulations.gov docket into a CSV file.

Description

This tool processes a directory containing comment JSON files from a regulations.gov docket and converts them into a single CSV file. It intelligently analyzes the data to include only meaningful fields, automatically excluding fields that are always null or have constant values across all files.

Features

• Smart Field Selection: Automatically excludes fields that are always null or have the same value across all files
• Flexible Output: Specify output directory, filename, or both
• Dry Run Mode: Preview which fields will be included/excluded without creating the CSV
• Force Include: Override automatic exclusion of constant-value fields
• Progress Tracking: Shows progress bars for large file sets
• Error Handling: Validates file structure and provides clear error messages

Requirements

• Python 3.9 or higher
• Required packages (see requirements.txt):
  • click
  • tqdm

Install

The setup.py works with pip to create a command mirrulations-csv. It is recommended that you create a virtual environment and install locally:

python3 -m venv .venv
source .venv/bin/activate
pip install .

Usage

Basic Usage

Convert all JSON files in a directory to CSV:

mirrulations-csv /path/to/json/files

Options

• -o, --output: Specify output file path or directory

  • mirrulations-csv /path/to/files -o output.csv
  • mirrulations-csv /path/to/files -o output/ (uses docket ID as filename)
  • mirrulations-csv /path/to/files -o output/results.csv

• --dryrun: Preview field analysis without creating CSV

  mirrulations-csv /path/to/files --dryrun

• -include: Force include a field that would normally be excluded

  mirrulations-csv /path/to/files -include fieldName

Examples

Preview what fields will be included:

mirrulations-csv DEA-2024-0059/raw-data/comments --dryrun

Convert to CSV with custom output location:

mirrulations-csv DEA-2024-0059/raw-data/comments -o results/

Force include a constant field:

mirrulations-csv DEA-2024-0059/raw-data/comments -include docketId