Varvis Download CLI

A command-line interface (CLI) tool for downloading BAM, BAI, and VCF files from the Varvis API. Built for bioinformatics workflows, it supports authentication, file filtering, proxy configuration, and archived file restoration.

Installation

Prerequisites

Node.js v22.0.0 or higher
NPM v10.0.0 or higher
External Tools (for range-based downloads):
- samtools v1.17+
- tabix v1.20+
- bgzip v1.20+

Setup

git clone https://github.com/LaborBerlin/varvis-download.git
cd varvis-download
npm install
chmod +x varvis-download.js
npm link  # Optional: makes the tool globally accessible

Quick Start

Basic Download

# Download all BAM/BAI files for specific analysis IDs
./varvis-download.js -u <username> -p <password> -t <target> -a <analysisId1,analysisId2>

# Using environment variables (recommended for security)
export VARVIS_USER="your_username"
export VARVIS_PASSWORD="your_password"
./varvis-download.js -t mytarget -a 12345,67890

List Available Files

# List files without downloading
./varvis-download.js -u <username> -p <password> -t <target> -a <analysisId> --list

Usage & Parameters

Required Parameters

Parameter	Short	Description	Example
`--target`	`-t`	API target	`mytarget`, `uni-leipzig`

At least one of the following is required:

--analysisIds (-a): Analysis IDs (comma-separated)
--sampleIds (-s): Sample IDs to filter analyses (comma-separated)
--limsIds (-l): LIMS IDs to filter analyses (comma-separated)

Authentication Options

Parameter	Short	Description	Note
`--username`	`-u`	Varvis API username	Can use `VARVIS_USER` env var
`--password`	`-p`	Varvis API password	Can use `VARVIS_PASSWORD` env var
`--config`	`-c`	Path to configuration file	Default: `.config.json`

File & Output Options

Parameter	Short	Default	Description
`--destination`	`-d`	`.`	Download destination folder
`--filetypes`	`-f`	`bam,bam.bai`	File types to download (comma-separated)
`--overwrite`	`-o`	`false`	Overwrite existing files
`--list`	`-L`	`false`	List available files without downloading

Filtering & Range Options

Parameter	Short	Description	Example
`--filter`	`-F`	Filter expressions (AND logic, multiple allowed)	`"enrichmentKitName^=TwistExome"`
`--latest`		Keep only newest analysis per sample	-
`--range`	`-g`	Genomic range	`"chr1:1-100000"`
`--bed`	`-b`	BED file with regions	`regions.bed`
`--unmapped`	`--um`	Extract unmapped reads from BAM	-

Filter operators: = != > < >= <= (lexicographic comparison), ~= (contains), ^= (starts with)

Archive Management

Parameter	Short	Default	Description
`--restoreArchived`	`-ra`	`ask`	Archive mode: `ask`, `all`, `force`, `no`
`--restorationFile`	`-rf`	`awaiting-restoration.json`	Restoration JSON file
`--resumeArchivedDownloads`	`-rad`	`false`	Resume archived downloads

Logging & Reports

Parameter	Short	Default	Description
`--loglevel`	`--ll`	`info`	Log level: `debug`, `info`, `warn`, `error`
`--logfile`	`--lf`	-	Path to log file
`--reportfile`	`-r`	-	Path to download report

Proxy Configuration

Parameter	Short	Description
`--proxy`	`-x`	Proxy URL
`--proxyUsername`	`--pxu`	Proxy username
`--proxyPassword`	`--pxp`	Proxy password

Other Options

Parameter	Short	Description
`--version`	`-v`	Show version information
`--help`	`-h`	Show help message

Authentication

Environment Variables (Recommended)

export VARVIS_USER="your_username"
export VARVIS_PASSWORD="your_password"
./varvis-download.js -t mytarget -a 12345

Configuration File

Create .config.json:

{
  "username": "your_username",
  "target": "mytarget",
  "destination": "./downloads",
  "loglevel": "info"
}

Interactive Password Prompt

If no password is provided via environment variables or CLI arguments, the tool will prompt for it securely with hidden input.

Archive Management

Archive Restoration Modes

Mode	Behavior
`ask` (default)	Prompt for each archived file
`all`	Ask once, apply to all archived files
`force`	Automatically restore all archived files
`no`	Skip archived files entirely

# Examples
./varvis-download.js -t mytarget -a 12345 --restoreArchived force
./varvis-download.js -t mytarget -a 12345 --restoreArchived no

Resume Archived Downloads

# Trigger restoration and save to custom file
./varvis-download.js -t mytarget -a 12345 --restoreArchived force --restorationFile my-restorations.json

# Later, resume downloads when files are ready
./varvis-download.js --resumeArchivedDownloads --restorationFile my-restorations.json

Advanced Features

Filtering Examples

# Filter by analysis type
./varvis-download.js -t mytarget -a 12345 -F "analysisType=SNV"

# Filter by enrichment kit (starts with)
./varvis-download.js -t mytarget -l "LIMS-001" -F "enrichmentKitName^=TwistExome"

# Multiple filters (AND logic) + keep only newest analysis per sample
./varvis-download.js -t mytarget -l "LIMS-001" -F "analysisType=SNV" -F "enrichmentKitName^=TwistExome" --latest

Range Downloads

# Single genomic range
./varvis-download.js -t mytarget -a 12345 -g "chr1:1-100000"

# Multiple ranges (space-separated)
./varvis-download.js -t mytarget -a 12345 -g "chr1:1-100000 chr2:1-100000"

# BED file with complex regions
./varvis-download.js -t mytarget -a 12345 -b complex-regions.bed

# Extract unmapped reads only
./varvis-download.js -t mytarget -a 12345 --unmapped

# Combined: genomic range + unmapped reads in a single BAM
./varvis-download.js -t mytarget -a 12345 -g "chr1:1-100000" --unmapped

Batch Operations

# Download multiple file types with custom settings
./varvis-download.js -t mytarget \
  -a "12345,67890,11111" \
  -f "bam,bam.bai,vcf.gz,vcf.gz.tbi" \
  -d "./batch-download" \
  --overwrite \
  --restoreArchived all

Proxy Usage

# Basic proxy
./varvis-download.js -t mytarget -a 12345 -x "http://proxy.example.com:8080"

# Proxy with authentication
./varvis-download.js -t mytarget -a 12345 -x "http://proxy.example.com:8080" --pxu proxy_user --pxp proxy_pass

Development

Quick Commands (Recommended)

Use Make for streamlined workflows:

make help           # Show all available commands
make dev            # Setup: install dependencies, lint, and test
make ci             # Run all quality checks (lint, format, type-check, test)

Testing

# Using Make (recommended)
make test           # Run all tests
make test-unit      # Unit tests only
make test-int       # Integration tests only

# Using npm
npm test                     # Run all tests
npm test -- --coverage       # With coverage report
npm run test:integration     # Integration tests

Code Quality

# Using Make (recommended)
make lint           # Check code quality
make lint-fix       # Auto-fix issues
make format         # Format with Prettier
make type-check     # Verify types

# Using npm
npm run lint        # Check code quality
npm run lint:fix    # Auto-fix ESLint issues
npm run format      # Format with Prettier
npm run type-check  # TypeScript type checking

Security

# Using Make (recommended)
make audit          # Check vulnerabilities (production only)
make audit-all      # Check all dependencies
make audit-fix      # Auto-fix vulnerabilities
make security       # Full security audit (npm + ESLint)

The project includes:

npm audit: Dependency vulnerability scanning
ESLint security rules: SAST for common security issues
Secret detection: Prevents secrets in code

Version Management

npm version patch   # Bug fixes: 0.21.0 → 0.21.1
npm version minor   # New features: 0.21.0 → 0.22.0
npm version major   # Breaking changes: 0.21.0 → 1.0.0

API Reference

Architecture Overview

graph TD;
    A[Main Function] --> B[AuthService.login]
    B --> C[AuthService.getCsrfToken]
    A --> D[fetchAnalysisIds]
    A --> E[getDownloadLinks]
    E --> F[Check Archive Status]
    A --> G[downloadFile]
    A --> H[generateReport]

Authentication Flow

sequenceDiagram
    participant User
    participant CLI
    participant AuthService
    participant VarvisAPI

    User->>CLI: Run with credentials
    CLI->>AuthService: login({username, password})
    AuthService->>VarvisAPI: Fetch CSRF token
    VarvisAPI-->>AuthService: Return CSRF token
    AuthService->>VarvisAPI: POST login with CSRF
    VarvisAPI-->>AuthService: Return session token
    AuthService-->>CLI: Authentication complete

    CLI->>VarvisAPI: Fetch analysis data
    CLI->>VarvisAPI: Download files
    CLI->>CLI: Generate report

Core Functions

`AuthService.login(user, target)`

Authenticates with Varvis API using CSRF tokens.

Parameters:

user: Object with username and password
target: Varvis API target (e.g., "mytarget")

Returns: Promise resolving to session token

`fetchAnalysisIds(target, token, agent, sampleIds, limsIds, filters, logger)`

Retrieves analysis IDs based on filtering criteria.

`getDownloadLinks(analysisId, filter, target, token, agent, logger, restoreArchived, rl)`

Fetches download links for files, handling archived file restoration.

`downloadFile(url, outputPath, overwrite, agent, rl, logger, metrics)`

Downloads a file with progress tracking and metrics collection.

Contributing

Contributions are welcome! Please:

Run make ci to verify all quality checks pass
Add tests for new functionality (all tests must pass)
Follow semantic versioning for API changes
Update documentation for new features

Development workflow:

make dev            # Setup and run initial checks
# ... make your changes ...
make lint-fix       # Auto-fix code quality issues
make ci             # Verify everything passes before committing

License

GPL-3.0 - See LICENSE file for details.

Name		Name	Last commit message	Last commit date
Latest commit History 128 Commits
.github		.github
.husky		.husky
assets		assets
docs		docs
js		js
tests		tests
.gitattributes		.gitattributes
.gitignore		.gitignore
.prettierignore		.prettierignore
LICENSE		LICENSE
Makefile		Makefile
README.md		README.md
eslint.config.js		eslint.config.js
jest.config.cjs		jest.config.cjs
jest.integration.config.cjs		jest.integration.config.cjs
jsdoc.conf.json		jsdoc.conf.json
make.cmd		make.cmd
package-lock.json		package-lock.json
package.json		package.json
tsconfig.json		tsconfig.json
varvis-download.cjs		varvis-download.cjs

Folders and files

Latest commit

History

Repository files navigation

Varvis Download CLI

Table of Contents

Installation

Prerequisites

Setup

Quick Start

Basic Download

List Available Files

Usage & Parameters

Required Parameters

Authentication Options

File & Output Options

Filtering & Range Options

Archive Management

Logging & Reports

Proxy Configuration

Other Options

Authentication

Environment Variables (Recommended)

Configuration File

Interactive Password Prompt

Archive Management

Archive Restoration Modes

Resume Archived Downloads

Advanced Features

Filtering Examples

Range Downloads

Batch Operations

Proxy Usage

Development

Quick Commands (Recommended)

Testing

Code Quality

Security

Version Management

API Reference

Architecture Overview

Authentication Flow

Core Functions

AuthService.login(user, target)

fetchAnalysisIds(target, token, agent, sampleIds, limsIds, filters, logger)

getDownloadLinks(analysisId, filter, target, token, agent, logger, restoreArchived, rl)

downloadFile(url, outputPath, overwrite, agent, rl, logger, metrics)

Contributing

License

About

Topics

Resources

License

Uh oh!

Stars

Watchers

Forks

Releases 2

Packages 0

Uh oh!

Uh oh!

Contributors

Uh oh!

Languages

`AuthService.login(user, target)`

`fetchAnalysisIds(target, token, agent, sampleIds, limsIds, filters, logger)`

`getDownloadLinks(analysisId, filter, target, token, agent, logger, restoreArchived, rl)`

`downloadFile(url, outputPath, overwrite, agent, rl, logger, metrics)`

Packages