feat!: v2.0.0 - Migrate to worker_threads architecture

[gkozlenko]

Summary: This PR radically overhauls the application's architecture by replacing the cluster module and multiple Express instances with a single Web worker and a pool of Media workers utilizing worker_threads. This approach significantly improves application state management and reduces inter-process communication overhead.

Key Changes:

• Worker Threads: Implemented a strict separation of concerns between web.js (handling HTTP requests) and the media.js pool (heavy media file processing). Cross-thread communication is handled directly via MessageChannel.
• Zero-Copy Memory Transfer: Passing generated HLS chunks and FragmentListDto structures between threads is now done via ArrayBuffer ownership transfer (transferables), eliminating the need to copy large datasets in memory.
• Refactoring & Cleanup: Removed obsolete components such as indexer.js, movie.js, and worker.js. DRM encryption logic has been extracted into a standalone cipher.js module.
• Configuration: Reworked and grouped the configuration into logical blocks (server, workers, web, logger, media).
• Dependencies: Updated key dependencies, including node-video-lib (v2.5.0), eslint (v10.0.1), and cors (v2.8.6).

Breaking Changes:

• The structure and keys in config/index.js have been completely redesigned.
• The server startup logic has changed: the application is now orchestrated by spawning worker threads from the main index.js process rather than forking child processes.

[gemini-code-assist]

Summary of Changes

Hello @gkozlenko, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces a major architectural shift, migrating the Node.js media server from a cluster-based approach to a worker_threads model. This refactoring enhances performance and maintainability by isolating web serving from media processing, optimizing inter-thread data transfer, and streamlining the application's configuration and component structure. The update also includes dependency upgrades and a clearer separation of concerns for encryption logic.

Highlights

• Architectural Migration to Worker Threads: The application's core architecture has been fundamentally redesigned, moving from Node.js cluster module and multiple Express instances to a single Web worker and a pool of Media workers using worker_threads. This change aims to improve state management and reduce inter-process communication overhead.
• Strict Separation of Concerns: The new architecture enforces a clear division between web.js (handling HTTP requests) and media.js (performing heavy media file processing). Communication between these threads is now managed directly via MessageChannel.
• Zero-Copy Memory Transfer: HLS chunks and FragmentListDto structures are now transferred between worker threads using ArrayBuffer ownership transfer (transferables), which eliminates the need for costly memory copying of large data sets.
• Refactoring and Cleanup: Obsolete components such as indexer.js, movie.js, and the generic worker.js have been removed. DRM encryption logic has been extracted into a new, dedicated cipher.js module.
• Configuration Rework: The config/index.js file has been restructured and grouped into logical blocks, including server, workers, web, logger, and media, to enhance clarity and maintainability.
• Dependency Updates: Key dependencies, including node-video-lib (updated to v2.5.0), eslint (updated to v10.0.1), and cors (updated to v2.8.6), have been updated to their latest versions.
• Breaking Changes: The configuration structure in config/index.js has been completely overhauled, and the server startup logic has changed significantly, now orchestrating worker threads from the main index.js process instead of forking child processes.

Changelog

• README.md
  • Updated project description to reflect 'Encryption' instead of 'DRM'.
  • Reworked the example configuration to align with the new grouped structure.
• components/cipher.js
  • Added a new module to encapsulate encryption key, IV generation, and chunk encryption logic.
• components/errors.js
  • Removed ForbiddenError and UnauthorizedError classes.
  • Added RequestTimeoutError, InternalServerError, and ServiceUnavailableError classes.
• components/indexer.js
  • Removed the dedicated indexer component as its functionality is now integrated into the media worker.
• components/logger.js
  • Updated logger configuration to use the new config.logger structure.
  • Modified log format to include stack traces for errors.
• components/media.js
  • Added a new component to manage media sessions, including creating and reading fragment lists and retrieving media chunks.
  • Implemented asynchronous file operations using fs/promises.
• components/movie.js
  • Removed the movie component, as its responsibilities are now handled by the media.js component and worker threads.
• components/worker.js
  • Removed the generic worker base class, as worker_threads provides direct messaging capabilities.
• config/index.js
  • Reworked the entire configuration structure, grouping settings into server, workers, web, logger, and media objects.
  • Updated paths and durations to reflect new configuration keys.
• index.js
  • Migrated from cluster module to worker_threads for managing application workers.
  • Implemented logic to spawn web and media worker threads and handle their lifecycle.
  • Updated shutdown procedures to gracefully terminate worker threads.
• models/fragment-list-dto.js
  • Added a new DTO (Data Transfer Object) for efficient transfer of fragment list metadata and buffer between worker threads.
• package-lock.json
  • Updated various dependency versions, including @eslint/js, eslint, cors, node-video-lib, acorn, ajv, balanced-match, brace-expansion, globals, minimatch, espree, and esquery.
  • Removed lodash and lodash.merge dependencies.
• package.json
  • Updated project version to 2.0.0.
  • Updated eslint and @eslint/js to v10.0.1.
  • Added globals as a dev dependency.
  • Updated cors to ^2.8.6 and node-video-lib to ^2.5.0.
  • Removed lodash dependency and overrides section for qs.
• routes/index.js
  • Removed lodash utility functions.
  • Simplified the SERVER_NAME format.
  • Removed specific error handling for ENOENT as it's now managed by HttpError.
• routes/vod/hls.js
  • Refactored HLS playlist and chunk serving to communicate with media worker threads.
  • Replaced direct file access and Movie component usage with req.callWorker for inter-thread communication.
  • Updated content types for HLS playlists.
  • Adapted encryption key retrieval to use the new cipher.js module and encryption.key endpoint.
• workers/indexer.js
  • Removed the dedicated indexer worker, as media indexing is now handled within the media.js worker.
• workers/media.js
  • Added a new worker thread responsible for media processing, including fragment list generation and chunk retrieval.
  • Implemented inter-thread communication with the web worker using MessageChannel and ArrayBuffer transferables.
  • Integrated media session management and encryption logic.
• workers/web.js
  • Rewritten as a worker_threads worker to host the Express application.
  • Implemented a request forwarding mechanism (req.callWorker) to communicate with media worker threads.
  • Added graceful shutdown logic for the web server and pending requests.
  • Updated access logging configuration to use the new config.logger paths.

Activity

• The pull request introduces a significant architectural change from cluster to worker_threads.
• New components like cipher.js, media.js, and fragment-list-dto.js have been added to support the new architecture.
• Obsolete components such as indexer.js, movie.js, and worker.js have been removed.
• Configuration has been extensively refactored for better organization.
• Dependencies have been updated to their latest versions, including major ESLint and node-video-lib upgrades.
• The core index.js and worker files (workers/web.js, workers/media.js) have been rewritten to implement the new worker thread model and inter-thread communication.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces a significant architectural shift from cluster to worker_threads, enhancing application state management and reducing IPC overhead. The changes are well-documented in the PR description, and the code reflects this migration by introducing new worker files and updating configuration. The removal of indexer.js, movie.js, and worker.js simplifies the codebase, and the new cipher.js module centralizes encryption logic. Dependency updates are also noted. Overall, the changes are substantial and appear to align with the stated goals of improving performance and maintainability.