This directory contains shell scripts and Python utilities organized by category to help with various system administration, data migration, monitoring, and document conversion tasks.
utilities/
├── analysis/ # Data analysis scripts
├── conversion/ # Document format conversion tools
├── migration/ # Data migration and recovery scripts
├── monitoring/ # System monitoring and alerting scripts
├── stornext/ # StorNext filesystem-specific utilities
├── system/ # System-level utilities (multipath, etc.)
└── testing/ # API and development testing scripts
Scripts for analyzing data, logs, and system information.
Analyzes date windows in archive files to identify missing time ranges. Examines add_date timestamps across multiple archive files and reports on entries within and outside specified time windows.
Usage:
python utilities/analysis/add_date_windows.pyPurpose: Identifies gaps in archive data by analyzing timestamp distributions across different archive files.
Analyzes missing file paths and their associated dm_info (data management info) blocks. Extracts and reports on flags, cpymap, class, vsn, totvers, and stub information for missing files.
Usage:
python utilities/analysis/analyze_missing_dm_info.pyPurpose: Helps diagnose why certain files are missing by examining their metadata characteristics.
Analyzes missing files and correlates them with media IDs, segment UUIDs, and add dates. Provides statistics on which media have the most missing files and calculates missing ratios per medium.
Usage:
python utilities/analysis/analyze_missing_media_stats.pyPurpose: Identifies problematic media or time periods that have high rates of missing files.
Tools for converting documents between different formats.
General-purpose PDF to Markdown converter. Extracts text from PDFs, removes page delimiters and footers, cleans up ligatures, and adds basic markdown formatting.
Usage:
python utilities/conversion/pdf_to_markdown.py input.pdf
python utilities/conversion/pdf_to_markdown.py input.pdf -o output.mdRequirements: pip install pypdf
Purpose: Converts PDF documents to markdown for easier editing, searching, and LLM processing.
Specialized converter for PDF man pages documentation. Extracts man pages from PDFs, adds proper markdown headings for commands and sections, and creates a structured document.
Usage:
python utilities/conversion/pdf_to_man_markdown.py input.pdf
python utilities/conversion/pdf_to_man_markdown.py input.pdf -o output.mdRequirements: pip install pypdf
Purpose: Converts PDF man page reference guides into searchable, structured markdown documentation.
Scripts for data migration, recovery, and bulk operations.
Generates a file list from a manifest and performs rsync operations from source to destination. Supports dry-run mode and handles missing file lists.
Usage:
utilities/migration/rsync_from_manifest.sh
utilities/migration/rsync_from_manifest.sh --dry-runConfiguration: Edit script to set SRC_ROOT, DEST_ROOT, MANIFEST, MISSING_LIST, and OUT_LIST paths.
Purpose: Performs efficient file synchronization based on manifest files, useful for large-scale data migrations.
Executes a command template against each directory listed in a file. Uses {} as a placeholder for the directory path.
Usage:
utilities/migration/run_on_dirs.sh 'fsretrieve -R "{}"'
utilities/migration/run_on_dirs.sh 'du -sh "{}"'
utilities/migration/run_on_dirs.sh 'ls -la "{}"'Configuration: Edit script to set BASE_PATH and DIRS_FILE paths.
Purpose: Batch operations on multiple directories, useful for applying commands across many paths.
Scripts for system monitoring and alerting.
Generates network throughput reports using sar and emails them. Designed for cron job scheduling.
Usage:
utilities/monitoring/network_throughput_email.sh user@example.comCron Example:
*/10 * * * * /path/to/utilities/monitoring/network_throughput_email.sh admin@example.comPurpose: Automated network performance monitoring with email alerts.
Executes a command and sends email notifications at start and completion. Captures stdout and stderr, compresses logs, and attaches them to completion email.
Usage:
-
Edit the script to configure:
COMMAND: The command to executeEMAIL_TO: Recipient email addressEMAIL_FROM: Sender email addressOUTPUT_DIR: Directory for log files
-
Run the script:
utilities/monitoring/run_with_email.shPurpose: Long-running command monitoring with email notifications and log archiving.
Utilities specific to StorNext filesystem operations.
Analyzes StorNext buffer cache efficiency by examining cvdb traces. Detects wasted buffer cache reads caused by excessive cachebuffersize mount settings.
Usage:
# Set environment variables (optional)
export TRACE_DIR=/tmp/cvdb_buffer_analysis
export DURATION=60
export OUTPUT_FILE=buffercache_waste_analysis.out
# Run the script
sudo utilities/stornext/analyze_buffercache_waste.shPurpose: Optimizes StorNext cachebuffersize mount option by identifying buffer cache waste.
Analyzes StorNext read I/O patterns using cvdb and snseq. Helps troubleshoot performance issues by examining sequential vs random reads, I/O sizes, latency, and fragmentation.
Usage:
# Set environment variables (optional)
export PARENT_DIR=/stornext/path/to/analyze
export DURATION=60
export OUTPUT_FILE=read_io_analysis.out
# Run the script
sudo utilities/stornext/analyze_read_io_patterns.shPurpose: Performance troubleshooting for StorNext read operations.
Recreates a directory tree structure with all metadata (permissions, ownership, timestamps, ACLs, extended attributes) from source to destination. Does NOT copy files - only directories and their metadata.
Usage:
utilities/stornext/clone_directory_tree.sh /source/dir /dest/dir
utilities/stornext/clone_directory_tree.sh -v -a -x /source/dir /dest/dir
utilities/stornext/clone_directory_tree.sh -d -v /source/dir /dest/dir # Dry runOptions:
-v, --verbose: Show detailed progress-d, --dry-run: Preview without making changes-a, --acls: Copy ACLs (requires getfacl/setfacl)-x, --xattrs: Copy extended attributes-l, --log FILE: Log output to file
Purpose: Preserves directory structure and metadata when migrating or backing up StorNext filesystems.
Recursively retrieves files from a managed StorNext directory to an unmanaged location. Creates a copy of files in a non-managed directory, preserving directory structure.
Usage:
utilities/stornext/fsretrieve_to_unmanaged.sh /stornext/managed/project1 /local/backup/project1
utilities/stornext/fsretrieve_to_unmanaged.sh -v -p 8 -c 1 /stornext/managed/archive /backup/archiveOptions:
-v, --verbose: Show detailed progress-d, --dry-run: Preview without making changes-p, --parallel N: Number of parallel retrieve operations (default: 4)-c, --copy N: Retrieve specific copy number (1, 2, etc.)-g, --glacier TYPE: Glacier restore type: standard, expedited, bulk-f, --force: Overwrite existing files-l, --log FILE: Log output to file
Purpose: Extracts files from StorNext managed storage to unmanaged locations for backup or migration.
System-level utilities for device management and recovery.
Quick reload of multipath device to fix blk_cloned_rq_check_limits errors. Minimal version that just reloads the device table.
Usage:
sudo utilities/system/quick_reload_multipath.sh dm-4Purpose: Fast fix for multipath device errors without full recovery procedure.
Recovery script for multipath device max_segment_kb issues. Fixes "blk_cloned_rq_check_limits: over max size limit" errors by adjusting device mapper table settings.
Usage:
sudo utilities/system/recover_multipath_max_segment.shPurpose: Comprehensive recovery for multipath devices experiencing segment size limit errors.
Reloads multipath device table without unmounting filesystems. Fixes "blk_cloned_rq_check_limits" errors by reloading the device mapper table while keeping filesystems mounted.
Usage:
sudo utilities/system/reload_multipath_without_unmount.shPurpose: Safe multipath device recovery without service interruption.
Development and API testing utilities.
Diagnostic script to test Anthropic API key configuration and identify issues. Checks environment variables, validates key format, and makes test API calls.
Usage:
export ANTHROPIC_API_KEY='your-key-here'
python utilities/testing/test_anthropic_api.pyPurpose: Validates Anthropic API key setup and troubleshoots authentication issues.
Direct API key testing tool that prompts for a key. Useful for testing keys without setting environment variables.
Usage:
python utilities/testing/test_anthropic_direct.py
# Paste your API key when promptedPurpose: Quick API key validation without environment variable setup.
Many scripts require root privileges, especially:
- StorNext scripts (cvdb commands)
- System scripts (multipath device management)
- Some migration scripts (file operations)
Python scripts may require additional packages:
pypdf- For PDF conversion scriptsanthropic- For API testing scripts
Install with:
pip install pypdf anthropicSeveral scripts use hardcoded paths or require environment variables. Check each script's header comments for configuration options.
Most scripts include error handling and cleanup routines. Check exit codes and log files for troubleshooting.
When adding new scripts:
- Place them in the appropriate subdirectory
- Add documentation to this README
- Include usage examples and requirements
- Add proper error handling and cleanup
Last updated: January 2026