`moducomp`: metabolic module completeness of genomes and metabolic complementarity in microbiomes

moducomp is a bioinformatics pipeline designed to identify and analyze metabolic module completeness and complementarity in microbial communities. It processes genomic data (protein sequences in FAA format) to map KEGG Orthology (KO) terms to KEGG modules, calculates module completeness for individual genomes and combinations of genomes within a microbiome, specifically reports potential complementary relationships where a metabolic module is 100% complete in a combination of N genomes but not in any individual member or smaller subset.

Features

Annotation of protein sequences using eggNOG-mapper to obtain KO terms.
Mapping of KOs to KEGG metabolic modules based on kegg-pathways-completeness-tool to obtain metabolic module completeness.
Parallel processing support for faster KPCT (KEGG Pathways Completeness Tool) analysis with automatic chunking and checkpointing.
Reporting of module completeness for individual genomes.
Calculation of module completeness for N-member genome combinations.
Generation of complementarity reports highlighting modules completed through genome partnerships.
Tracks and reports the actual proteins that are responsible for the completion of the module in the combination of N genomes.
Automatic resource monitoring with timestamped logs tracking CPU usage, memory consumption, and runtime for reproducibility.
Consistent logging to stdout/stderr with a per-command resource summary emitted at the end of each run.
Built-in validation (moducomp validate) for scientific consistency checks across annotations, KO matrices, KPCT outputs, and complementarity reports.

Installation (Recommended)

Make sure you have Pixi installed:

curl -fsSL https://pixi.sh/install.sh | sh

Install moducomp with Pixi:

pixi global install \
  -c conda-forge \
  -c bioconda \
  -c https://repo.prefix.dev/astrogenomics \
  moducomp

Setup data (required)

moducomp needs the eggNOG-mapper database to run. Use moducomp setup to download the data using ModuComp's built-in downloader (with resume/retry) and record the location for future runs. For upstream details, see the eggNOG-mapper setup guide: eggNOG-mapper database setup.

moducomp setup

By default, the data are stored in ${XDG_DATA_HOME:-~/.local/share}/moducomp/eggnog, and moducomp will auto-detect that location on future runs without needing EGGNOG_DATA_DIR.

To use a custom location (or point to an existing download):

moducomp setup --eggnog-data-dir /path/to/eggnog-data

Add --force to re-download the data if the directory already exists. Use --prompt if you want to confirm each download interactively.

Quick test

Small test data sets ship with moducomp. After installation you can confirm the pipeline by running:

moducomp test --ncpus 16 --calculate-complementarity 2

The test command runs in low-memory mode by default. If you have plenty of RAM and want full-memory mode, add --fullmem (or --full-mem).

Developer install (Pixi)

If you want to download the code and develop locally:

git clone https://github.com/NeLLi-team/moducomp.git
cd moducomp
pixi install

Quick check

moducomp --help

If you are running from the repository with Pixi:

pixi run python -m moducomp --help

You should see the command line help without errors.

Usage

moducomp provides two main commands: pipeline and analyze-ko-matrix. You can run these commands using Pixi tasks defined in pyproject.toml or directly within the Pixi environment.

Pipeline overview

The diagram below shows the main stages executed by ModuComp.

graph TD
    A([Start run]) --> B[Initialize logging and resource monitoring]
    B --> C{Input type}
    C -->|pipeline| D[Validate genome directory]
    C -->|analyze-ko-matrix| H[Load existing KO matrix]
    D --> E[Prepare genomes: adapt headers or copy to tmp]
    E --> F[Merge genomes into single FAA]
    F --> G[Run eggNOG-mapper if needed]
    G --> H[Create KO matrix kos_matrix.csv]
    H --> I[Convert KO matrix to KPCT input]
    I --> J[Run KPCT with parallel fallback]
    J --> K[Create module completeness matrix]
    K --> L{Complementarity requested?}
    L -->|Yes| M[Generate complementarity reports]
    L -->|No| N[Skip]
    M --> O[Write outputs and logs]
    N --> O
    O --> P[Optional cleanup of tmp]
    P --> Q([Pipeline complete])

CLI options and defaults

This section lists all CLI options implemented today, along with their default values.

`pipeline` command (positional args: `genomedir`, `savedir`)

Option	Default	Description
`--ncpus`, `-n`	`16`	Number of CPU cores to use for eggNOG-mapper and KPCT.
`--calculate-complementarity`, `-c`	`0`	Complementarity size to compute (0 disables).
`--adapt-headers/--no-adapt-headers`	`false`	Adapt FASTA headers to `genome
`--del-tmp/--keep-tmp`	`true`	Delete temporary files after completion.
`--lowmem/--fullmem` (`--low-mem/--full-mem`)	`lowmem`	Run eggNOG-mapper without `--dbmem` to reduce RAM.
`--verbose/--quiet`	`false`	Enable verbose progress output.
`--validate/--no-validate`	`validate`	Run post-run validation checks.
`--validate-report/--no-validate-report`	`validate-report`	Write `validation_report.json` in the output directory.
`--validate-strict/--validate-lenient`	`lenient`	Treat validation warnings as failures when strict.
`--log-level`, `-l`	`INFO`	Logging level: `DEBUG`, `INFO`, `WARNING`, `ERROR`.
`--eggnog-data-dir`	`EGGNOG_DATA_DIR`	Path to eggNOG-mapper data (sets `EGGNOG_DATA_DIR`).

`test` command (bundled test genomes)

Option	Default	Description
`--output-dir`, `-o`	`output_test_moducomp_<DATETIME>`	Output directory for test run.
`--ncpus`, `-n`	`2`	CPU cores for the test run.
`--calculate-complementarity`, `-c`	`2`	Complementarity size to compute (0 disables).
`--adapt-headers/--no-adapt-headers`	`false`	Adapt FASTA headers before the test.
`--del-tmp/--keep-tmp`	`true`	Delete temporary files after the test completes.
`--lowmem/--fullmem` (`--low-mem/--full-mem`)	`lowmem`	Low-memory mode is the default for tests.
`--verbose/--quiet`	`verbose`	Verbose output is the default for tests.
`--validate/--no-validate`	`validate`	Run post-run validation checks.
`--validate-report/--no-validate-report`	`validate-report`	Write `validation_report.json` in the output directory.
`--validate-strict/--validate-lenient`	`lenient`	Treat validation warnings as failures when strict.
`--log-level`, `-l`	`INFO`	Logging level: `DEBUG`, `INFO`, `WARNING`, `ERROR`.
`--eggnog-data-dir`	`EGGNOG_DATA_DIR`	Path to eggNOG-mapper data (sets `EGGNOG_DATA_DIR`).

`analyze-ko-matrix` command (positional args: `kos_matrix`, `savedir`)

Option	Default	Description
`--calculate-complementarity`, `-c`	`0`	Complementarity size to compute (0 disables).
`--kpct-outprefix`	`output_give_completeness`	Prefix for KPCT output files.
`--del-tmp/--keep-tmp`	`true`	Delete temporary files after completion.
`--ncpus`, `-n`	`16`	CPU cores for KPCT parallel processing.
`--verbose/--quiet`	`false`	Enable verbose progress output.
`--validate/--no-validate`	`validate`	Run post-run validation checks.
`--validate-report/--no-validate-report`	`validate-report`	Write `validation_report.json` in the output directory.
`--validate-strict/--validate-lenient`	`lenient`	Treat validation warnings as failures when strict.
`--log-level`, `-l`	`INFO`	Logging level: `DEBUG`, `INFO`, `WARNING`, `ERROR`.

`validate` command (positional args: `savedir`)

Option	Default	Description
`--mode`	`auto`	Validation mode: `auto`, `pipeline`, or `ko-matrix`.
`--calculate-complementarity`, `-c`	`auto-detect`	Expected complementarity size (0 disables).
`--kpct-outprefix`	`output_give_completeness`	KPCT output prefix used during analysis.
`--strict/--lenient`	`lenient`	Treat warnings as failures when strict.
`--report`	none	Write JSON validation report to this path.
`--verbose/--quiet`	`false`	Enable verbose progress output.
`--log-level`, `-l`	`INFO`	Logging level: `DEBUG`, `INFO`, `WARNING`, `ERROR`.

`setup` command

Option	Default	Description
`--eggnog-data-dir`	`${XDG_DATA_HOME:-~/.local/share}/moducomp/eggnog`	Destination for eggNOG-mapper data (sets `EGGNOG_DATA_DIR`).
`--yes/--prompt`, `-y`	`yes`	Automatically accept core database downloads (use `--prompt` to confirm each).
`--force`, `-f`	`false`	Re-download data even if files already exist.
`--log-level`, `-l`	`INFO`	Logging level: `DEBUG`, `INFO`, `WARNING`, `ERROR`.
`--verbose/--quiet`	`verbose`	Stream downloader output to the console.

Performance and parallel processing

moducomp includes parallel processing capabilities for the KPCT (KEGG Pathways Completeness Tool) analysis, which can significantly improve performance for large datasets:

Automatic chunking: Input files are automatically split into chunks for parallel processing
Checkpointing: Resume capability if processing is interrupted - already processed chunks are automatically detected and skipped
Fallback mechanism: If parallel processing fails, the system automatically falls back to sequential processing
Configurable CPU usage: Use the --ncpus parameter to control how many CPU cores to use

CPU Configuration:

The --ncpus parameter controls the number of CPU cores used for both eggNOG-mapper annotation and KPCT analysis
For KPCT parallel processing, the system creates the same number of chunks as CPU cores specified
Example: --ncpus 8 will use 8 cores and create 8 chunks for optimal parallel processing

Validation (QC)

Use the built-in validator to check scientific consistency across outputs after a run. The validator compares:

KO sets and counts between eggNOG-mapper annotations and kos_matrix.csv
KO sets between kos_matrix.csv and ko_file_for_kpct.txt
KPCT contigs vs pathways outputs
Module completeness ranges and combination naming
Complementarity reports versus module completeness values
Protein provenance fields (pipeline mode) or placeholders (KO-matrix mode)

Example:

# Validation runs by default after pipeline/analyze/test.
# Use --no-validate to disable or --no-validate-report to skip JSON output.
# When validation reports errors (or warnings in strict mode), the command exits non-zero.

# Validate a pipeline run and write a JSON report
moducomp validate /path/to/output --mode pipeline --report /path/to/output/validation_report.json

# Validate KO-matrix mode outputs (non-default KPCT prefix)
moducomp validate /path/to/output --mode ko-matrix --kpct-outprefix my_prefix

# Treat warnings as failures
moducomp validate /path/to/output --strict

⚠️ Important note 1

Prepare FAA files: Ensure FAA headers are in the form >genomeName|proteinId, or use the --adapt-headers option to format your headers into >fileName_prefix|protein_id_counter.

⚠️ Important note 2

moducomp is specifically designed for large scale analysis of microbiomes with hundreds of members, and works on Linux systems with at least 64GB of RAM. For robustness, low-memory mode is now the default for pipeline and test. If you have ample RAM and want full-memory mode, add --fullmem (--full-mem).

Notes on bundled test data

You can override the bundled data location with MODUCOMP_DATA_DIR. When working from source, the bundled test genomes live at moducomp/data/test_genomes.

Use moducomp setup to download and register the eggNOG data directory. The location is stored in ${XDG_CONFIG_HOME:-~/.config}/moducomp/config.json so future runs can find the data automatically.

From source (Pixi):

pixi run python -m moducomp setup --eggnog-data-dir /path/to/eggnog-data

Running with your samples

If you are running from the repository with Pixi, replace moducomp below with pixi run python -m moducomp.

`pipeline` command

Use the pipeline command to process a directory of genome FAA files from scratch.

moducomp pipeline \
    /path/to/your/faa_files \
    /path/to/your/output_directory \
    --ncpus <number_of_cpus_to_use> \
    --calculate-complementarity <N>  # 0 to disable, 2 for 2-member, 3 for 3-member complementarity.
    # Optional flags:
    # --fullmem                  # Optional: Use full-mem if you have ample RAM (default is low-mem)
    # --adapt-headers             # If your FASTA headers need modification
    # --del-tmp/--keep-tmp        # Delete or keep temporary files
    # --eggnog-data-dir /path     # If EGGNOG_DATA_DIR is not set
    # --verbose                   # Enable verbose output with detailed progress information

`analyze-ko-matrix` command

Use the analyze-ko-matrix command if you already have a KO matrix file (CSV format, where rows are genomes/combinations, columns are KOs, and values are KO counts).

The KO matrix file should have a taxon_oid column for genome identifiers, and subsequent columns for each KO (e.g., K00001, K00002) with integer counts.

moducomp analyze-ko-matrix \
    /path/to/your/kos_matrix.csv \
    /path/to/your/output_directory \
    --ncpus <number_of_cpus_to_use> \
    --calculate-complementarity <N>  # 0 to disable, 2 for 2-member, 3 for 3-member complementarity.

    # Optional flags:
    # --keep-tmp                  # Keep temporary files
    # --verbose                   # Enable verbose output with detailed progress information

Parallel processing features

moducomp includes advanced parallel processing capabilities for improved performance:

KPCT parallel processing

When using the --ncpus parameter with a value greater than 1, moducomp automatically enables parallel processing for the KPCT (KEGG Pathways Completeness Tool) analysis:

Automatic chunking: Input files are split into ncpus chunks for optimal load balancing
Concurrent processing: Multiple chunks are processed simultaneously using multiprocessing.ProcessPoolExecutor
Resume capability: If processing is interrupted, completed chunks are automatically detected and skipped on restart
Automatic fallback: If parallel processing fails, the system seamlessly falls back to sequential processing

Performance tips

CPU cores: Start with --ncpus 8 for moderate datasets, increase to --ncpus 16 or higher for large datasets
Memory considerations: Each parallel worker requires memory; reduce --ncpus if you encounter memory issues
Large datasets: For datasets with hundreds of genomes, parallel processing can reduce KPCT analysis time by 50-80%

Example with parallel processing

# For large datasets with sufficient resources
moducomp pipeline ./large_genome_collection ./output_large --ncpus 32 --calculate-complementarity 3

# For moderate datasets with verbose output
moducomp analyze-ko-matrix ./ko_matrix.csv ./output_moderate --ncpus 16 --calculate-complementarity 2 --verbose

# For systems with limited memory (default behavior)
moducomp pipeline ./genomes ./output_lowmem --ncpus 8 --calculate-complementarity 2

# For systems with ample RAM
moducomp pipeline ./genomes ./output_fullmem --ncpus 8 --fullmem --calculate-complementarity 2

Expected outputs

The sections below describe the expected output files, naming conventions, and the column-level meaning of each file. These details are the same for moducomp pipeline and moducomp test (pipeline mode), and the subset noted for moducomp analyze-ko-matrix (KO-matrix mode).

Naming conventions

Genome identifiers are stored as taxon_oid. In pipeline mode, ModuComp expects protein headers in the format genome_id|protein_id. If you set --adapt-headers, ModuComp rewrites headers to >genomeName|protein_N, where genomeName is the FAA filename stem. Combination identifiers use __ (double underscore), for example GenomeA__GenomeB, and n_members in module_completeness.tsv records the size of each combination.

Pipeline mode outputs (moducomp pipeline, moducomp test)

emapper_out.emapper.annotations: Full eggNOG-mapper annotations. The #query column must match genome_id|protein_id. KEGG_ko entries are prefixed ko:KXXXXX and are converted to KXXXXX for downstream matrices.
kos_matrix.csv: Genome × KO count matrix. Columns: taxon_oid followed by KO IDs (e.g., K00001). Values are integer protein counts per KO.
ko_file_for_kpct.txt: KPCT input file. Each line starts with taxon_oid followed by the set of KO IDs present in that genome or combination. If --calculate-complementarity is N>=2, combinations up to N are included as GenomeA__GenomeB.
output_give_completeness_contigs.with_weights.tsv: KPCT module results per genome/combination. Columns: contig (genome/combination ID), module_accession, completeness (0–100), pathway_name, pathway_class, matching_ko (KO weights), missing_ko.
output_give_completeness_pathways.with_weights.tsv: Same rows and order as the contigs file, but without the contig column. This is provided for compatibility with legacy tools; prefer the contigs file when you need genome-level provenance.
module_completeness.tsv: Pivoted module completeness matrix. Columns: n_members, taxon_oid, followed by KEGG module IDs (M00001, …). Values are numeric percentages in the range 0–100.
module_completeness_complementarity_Nmember.tsv: Complementarity report for N-member combinations (only when --calculate-complementarity N is set). Columns: taxon_oid_1..N, completeness_taxon_oid_1..N, module_id, module_name, pathway_class, matching_ko, proteins_taxon_oid_1..N. Protein fields list contributing proteins per KO (from eggNOG-mapper) as {'KXXXXX': 'genome|protein'}.
logs/moducomp.log: Detailed run log with structured progress messages and per-command resource summaries.
logs/resource_usage_YYYYMMDD_HHMMSS.log: Resource monitoring log capturing wall time, CPU time, CPU utilization, peak RAM, and exit code for each monitored command.
tmp/ (only if --keep-tmp): Intermediate files such as merged_genomes.faa, emapper_output/, and KPCT chunk outputs.
validation_report.json (default when validation is enabled): JSON report produced by the validator.

KO-matrix mode outputs (moducomp analyze-ko-matrix)

kos_matrix.csv: A copy of the input KO matrix (same format as above).
ko_file_for_kpct.txt: KPCT input generated from the KO matrix. If --calculate-complementarity is set, combination lines are added using GenomeA__GenomeB identifiers.
output_give_completeness_contigs.with_weights.tsv: KPCT module results per genome/combination (same format as pipeline mode).
output_give_completeness_pathways.with_weights.tsv: Same rows as the contigs file, without the contig column.
module_completeness.tsv: Module completeness matrix (same format as pipeline mode).
module_completeness_complementarity_Nmember.tsv: Complementarity report. Protein contribution columns are filled with No protein data available for <genome> because no eggNOG-mapper annotations are available in KO-matrix mode.
logs/moducomp.log and logs/resource_usage_YYYYMMDD_HHMMSS.log: Standard run logs and resource summaries.
validation_report.json (default when validation is enabled): JSON report produced by the validator.

Citation

Villada, JC. & Schulz, F. (2025). Assessment of metabolic module completeness of genomes and metabolic complementarity in microbiomes with moducomp . moducomp (v0.5.1) Zenodo. https://doi.org/10.5281/zenodo.16116092

Tool Citations

If you use ModuComp in a study, please also cite the wrapped tools and data sources:

eggNOG-mapper: Cantalapiedra CP et al. (2021). “eggNOG-mapper v2: Functional Annotation, Orthology Assignments, and Domain Prediction at the Metagenomic Scale.” Molecular Biology and Evolution. DOI: 10.1093/molbev/msab293. GitHub: https://github.com/eggnogdb/eggnog-mapper
KEGG Pathways Completeness Tool (KPCT): GitHub: https://github.com/EBI-Metagenomics/kegg-pathways-completeness-tool. KEGG source: Kanehisa M, Goto S. (2000). “KEGG: Kyoto Encyclopedia of Genes and Genomes.” Nucleic Acids Research. DOI: 10.1093/nar/28.1.27

Name		Name	Last commit message	Last commit date
Latest commit History 59 Commits
moducomp		moducomp
.gitignore		.gitignore
LICENSE.txt		LICENSE.txt
README.md		README.md
pixi.lock		pixi.lock
pyproject.toml		pyproject.toml
recipe.yaml		recipe.yaml

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

`moducomp`: metabolic module completeness of genomes and metabolic complementarity in microbiomes

Features

Installation (Recommended)

Setup data (required)

Quick test

Developer install (Pixi)

Quick check

Usage

Pipeline overview

CLI options and defaults

`pipeline` command (positional args: `genomedir`, `savedir`)

`test` command (bundled test genomes)

`analyze-ko-matrix` command (positional args: `kos_matrix`, `savedir`)

`validate` command (positional args: `savedir`)

`setup` command

Performance and parallel processing

Validation (QC)

⚠️ Important note 1

⚠️ Important note 2

Notes on bundled test data

Running with your samples

`pipeline` command

`analyze-ko-matrix` command

Parallel processing features

KPCT parallel processing

Performance tips

Example with parallel processing

Expected outputs

Citation

Tool Citations

About

Uh oh!

Releases 18

Packages

Languages

License

NeLLi-team/moducomp

Folders and files

Latest commit

History

Repository files navigation

moducomp: metabolic module completeness of genomes and metabolic complementarity in microbiomes

Features

Installation (Recommended)

Setup data (required)

Quick test

Developer install (Pixi)

Quick check

Usage

Pipeline overview

CLI options and defaults

pipeline command (positional args: genomedir, savedir)

test command (bundled test genomes)

analyze-ko-matrix command (positional args: kos_matrix, savedir)

validate command (positional args: savedir)

setup command

Performance and parallel processing

Validation (QC)

⚠️ Important note 1

⚠️ Important note 2

Notes on bundled test data

Running with your samples

pipeline command

analyze-ko-matrix command

Parallel processing features

KPCT parallel processing

Performance tips

Example with parallel processing

Expected outputs

Citation

Tool Citations

About

Resources

License

Uh oh!

Stars

Watchers

Forks

Releases 18

Packages 0

Languages

`moducomp`: metabolic module completeness of genomes and metabolic complementarity in microbiomes

`pipeline` command (positional args: `genomedir`, `savedir`)

`test` command (bundled test genomes)

`analyze-ko-matrix` command (positional args: `kos_matrix`, `savedir`)

`validate` command (positional args: `savedir`)

`setup` command

`pipeline` command

`analyze-ko-matrix` command

Packages