DevelopRelease v0.9.3 – Canonical Authority & Scheduler Hardening

.env.example

@@ -1,3 +1,28 @@
 RETREIVR_DATA_DIR=/app/data  # App data (SQLite/temp) root; change only if you use a different internal mount point.
 RETREIVR_HOST=0.0.0.0  # Bind address for the API; keep 0.0.0.0 in containers for external access.
 RETREIVR_PORT=8000  # Internal API port; change only if you modify container port mapping.
+
+# --- Storage ---
+RETREIVR_DOWNLOADS_DIR=/app/downloads  # Root directory for downloaded media inside container.
+
+# --- Video Container Policy (v0.9.3 default = mkv) ---
+RETREIVR_DEFAULT_VIDEO_FORMAT=mkv  # Options: mkv | mp4 | webm (mkv recommended for archival fidelity).
+
+# --- Scheduler ---
+RETREIVR_SCHEDULER_INTERVAL_MINUTES=30  # Interval for playlist/watch polling.
+
+# --- Spotify (Optional – requires OAuth + Premium) ---
+SPOTIFY_CLIENT_ID=
+SPOTIFY_CLIENT_SECRET=
+SPOTIFY_REDIRECT_URI=http://localhost:8090/api/spotify/callback
+
+# Spotify functionality is disabled unless OAuth is configured AND Premium validation succeeds.
+
+# --- Telegram Notifications (Optional) ---
+TELEGRAM_BOT_TOKEN=
+TELEGRAM_CHAT_ID=
+
+# Leave empty to disable Telegram notifications.
+
+# --- Logging ---
+RETREIVR_LOG_LEVEL=INFO  # Options: DEBUG | INFO | WARNING | ERROR

CHANGELOG.md

@@ -2,6 +2,73 @@
 
 All notable changes to this project will be documented here.
 
+## [v0.9.3] – Canonical Authority & Scheduler Hardening
+
+This release establishes Retreivr’s canonical authority model and locks in deterministic orchestration behavior. v0.9.3 is a stability milestone focused on correctness, idempotency, and clean archival output.
+
+### Highlights
+
+- MusicBrainz is now the canonical metadata authority.
+- Spotify downgraded to optional intent ingestion (OAuth + Premium required).
+- Spotify API usage (playlists, saved tracks, metadata hints) now strictly requires OAuth configuration and an active Premium subscription.
+- Deterministic playlist snapshot hashing and diffing.
+- Idempotent scheduler ticks (no duplicate enqueues).
+- MKV set as the default video container.
+- Integration tests added for full pipeline and snapshot behavior.
+
+---
+
+### Added
+
+- Structured `PlaylistRunSummary` with:
+  - `added`
+  - `skipped`
+  - `completed`
+  - `failed`
+- Stable playlist snapshot hashing using normalized item sets.
+- Crash-safe restart behavior for scheduler runs.
+- Active-job duplicate detection (queued / claimed / downloading / postprocessing states).
+- Integration tests covering:
+  - Full music flow (search → resolve → download → embed → persist)
+  - Spotify intent conversion (MB-first enforcement)
+  - Playlist reorder behavior (no re-enqueue)
+  - Crash/restart idempotency
+- MKV default container policy for video downloads.
+
+---
+
+### Changed
+
+- Canonical metadata resolution is now MusicBrainz-first in all ingestion paths.
+- Spotify metadata is treated as hints only and never overrides MusicBrainz canonical results.
+- Legacy resolver paths removed.
+- Duplicate MusicBrainz client stacks consolidated into a single service layer.
+- Canonical naming enforced:
+  - No video IDs in filenames
+  - No upload dates in filenames
+  - Zero-padded music track numbers
+- Video metadata embedding now occurs after final container merge, ensuring metadata survives remux.
+- Scheduler diff logic hardened to ignore reorder-only changes.
+- Snapshot persistence made deterministic to prevent unnecessary DB churn.
+
+---
+
+### Fixed
+
+- Prevented duplicate active-job enqueue on scheduler restart.
+- Eliminated reorder-triggered playlist re-downloads.
+- Fixed snapshot instability caused by unordered playlist items.
+- Prevented metadata failures from corrupting or blocking completed downloads.
+
+---
+
+### Notes
+
+- This release prioritizes stability over new feature expansion.
+- v0.9.3 marks the transition to a canonical, deterministic ingestion engine.
+- MKV is now the default video container to preserve codec fidelity and improve metadata support.
+- Spotify integration depends on the official Spotify Web API and requires valid OAuth credentials plus Premium account validation; without these, Spotify playlist sync and metadata ingestion remain disabled.
+
 ## [v0.9.2] – Search Engine Dialed In // Home Page UI Update
 
 Highlights
@@ -10,89 +77,57 @@ This release hardens the download pipeline (especially audio-only MP3), improves
 
 ⸻
 
-🚀 Improvements
-	•	Reliable MP3 audio-only downloads
-	•	Audio mode now uses a robust bestaudio[acodec!=none]/bestaudio/best selector.
-	•	Prevents unnecessary video downloads when targeting MP3.
-	•	Matches known-working yt-dlp CLI behavior.
-	•	Works consistently for direct URLs and queued jobs.
-	•	Safer yt-dlp option handling
-	•	Avoids forced merge/remux unless explicitly required.
-	•	Reduces ffmpeg post-processing failures.
-	•	Audio and video paths are now clearly separated and predictable.
-	•	yt-dlp CLI observability
-	•	Job workers now log the exact yt-dlp CLI command executed (with secrets redacted).
-	•	Makes debugging format, cookie, and extractor issues significantly easier.
+🚀 Improvements • Reliable MP3 audio-only downloads • Audio mode now uses a robust bestaudio[acodec!=none]/bestaudio/best selector. • Prevents unnecessary video downloads when targeting MP3. • Matches known-working yt-dlp CLI behavior. • Works consistently for direct URLs and queued jobs. • Safer yt-dlp option handling • Avoids forced merge/remux unless explicitly required. • Reduces ffmpeg post-processing failures. • Audio and video paths are now clearly separated and predictable. • yt-dlp CLI observability • Job workers now log the exact yt-dlp CLI command executed (with secrets redacted). • Makes debugging format, cookie, and extractor issues significantly easier.
 
 ⸻
 
-🧠 Behavior Fixes
-	•	Post-processing failures are now terminal
-	•	ffmpeg / post-processing errors correctly mark jobs as FAILED.
-	•	Prevents silent re-queue loops and misleading “Queued” states in the UI.
-	•	Video pipeline preserved
-	•	Default video behavior (bestvideo+bestaudio/best) remains unchanged.
-	•	MP4 / MKV / WebM downloads continue to work as before.
+🧠 Behavior Fixes • Post-processing failures are now terminal • ffmpeg / post-processing errors correctly mark jobs as FAILED. • Prevents silent re-queue loops and misleading “Queued” states in the UI. • Video pipeline preserved • Default video behavior (bestvideo+bestaudio/best) remains unchanged. • MP4 / MKV / WebM downloads continue to work as before.
 
 ⸻
 
-🎧 Music & Metadata
-	•	Music metadata enrichment remains optional
-	•	Failed or low-confidence enrichment no longer blocks successful downloads.
-	•	Clear logging when metadata is skipped due to confidence thresholds.
+🎧 Music & Metadata • Music metadata enrichment remains optional • Failed or low-confidence enrichment no longer blocks successful downloads. • Clear logging when metadata is skipped due to confidence thresholds.
 
 ⸻
 
-🖥 UI / UX
-	•	Home page cleanup
-	•	Reorganized source filters and advanced options into a single compact row.
-	•	Reduced visual noise without removing functionality.
-	•	Improved spacing and alignment for music mode, format, and destination controls.
-	•	Advanced Search remains available
-	•	Advanced functionality is still accessible via the dedicated Advanced Search page.
+🖥 UI / UX • Home page cleanup • Reorganized source filters and advanced options into a single compact row. • Reduced visual noise without removing functionality. • Improved spacing and alignment for music mode, format, and destination controls. • Advanced Search remains available • Advanced functionality is still accessible via the dedicated Advanced Search page.
 
 ⸻
 
-🧹 Internal / Maintenance
-	•	Improved internal option auditing logs.
-	•	Better separation between search, enqueue, and execution logic.
-	•	No schema or config migrations required.
+🧹 Internal / Maintenance • Improved internal option auditing logs. • Better separation between search, enqueue, and execution logic. • No schema or config migrations required.
 
 ⸻
 
-⚠️ Known Notes
-	•	Client-side (“download to this device”) delivery is still being refined and may be disabled or hidden in some UI paths.
-
+⚠️ Known Notes • Client-side (“download to this device”) delivery is still being refined and may be disabled or hidden in some UI paths.
 
 ## [v0.9.1] – Runtime Stability & Direct URL Fixes
 
-This release focuses on restoring and hardening runtime stability after refactors since yt-archiver v1.2.0.
-Primary goals were correctness, predictability, and eliminating regressions in downloads, scheduling, and search flows.
+This release focuses on restoring and hardening runtime stability after refactors since yt-archiver v1.2.0. Primary goals were correctness, predictability, and eliminating regressions in downloads, scheduling, and search flows.
 
 Fixed:
-- Restored reliable Direct URL downloads for video and audio (mp3/m4a/etc).
-- Corrected yt-dlp invocation for audio formats (uses extract-audio instead of merge-output-format).
-- Fixed Direct URL runs appearing permanently queued in the Home UI.
-- Prevented empty or zero-byte output files from being recorded as completed.
-- Fixed scheduler playlist downloads producing incorrect formats or audio-only output.
-- Ensured scheduler and direct downloads can run concurrently without interference.
-- Fixed missing database schema initialization for search-related tables.
-- Normalized all filesystem paths via paths.py and environment variables (Docker-safe).
-- Fixed Advanced Search “Failed to load requests” error caused by search DB store calling service-only logic.
-- Fixed Home screen results remaining stuck in “Queued” by restoring reliable search request status hydration.
-- Unified search job database usage to a single canonical path to prevent schema and state mismatches.
 
+Restored reliable Direct URL downloads for video and audio (mp3/m4a/etc).
+Corrected yt-dlp invocation for audio formats (uses extract-audio instead of merge-output-format).
+Fixed Direct URL runs appearing permanently queued in the Home UI.
+Prevented empty or zero-byte output files from being recorded as completed.
+Fixed scheduler playlist downloads producing incorrect formats or audio-only output.
+Ensured scheduler and direct downloads can run concurrently without interference.
+Fixed missing database schema initialization for search-related tables.
+Normalized all filesystem paths via paths.py and environment variables (Docker-safe).
+Fixed Advanced Search “Failed to load requests” error caused by search DB store calling service-only logic.
+Fixed Home screen results remaining stuck in “Queued” by restoring reliable search request status hydration.
+Unified search job database usage to a single canonical path to prevent schema and state mismatches.
 Changed:
-- Direct URL playlist links are now explicitly rejected with a clear user-facing error message.
-- Direct URL runs bypass the job queue but still report progress and completion via run status.
-- Search-only results can now be downloaded individually via the Home results UI.
-- Default video downloads respect configured format preferences (e.g., webm/mp4).
-- Metadata enrichment failures no longer block or corrupt completed downloads.
 
+Direct URL playlist links are now explicitly rejected with a clear user-facing error message.
+Direct URL runs bypass the job queue but still report progress and completion via run status.
+Search-only results can now be downloaded individually via the Home results UI.
+Default video downloads respect configured format preferences (e.g., webm/mp4).
+Metadata enrichment failures no longer block or corrupt completed downloads.
 Notes:
-- Playlist URLs must be added via Scheduler / Playlist configuration, not Direct URL mode.
-- Kill-download button is not guaranteed during active runs and remains experimental.
-- Watcher functionality is present but considered beta and may change in later releases.
+
+Playlist URLs must be added via Scheduler / Playlist configuration, not Direct URL mode.
+Kill-download button is not guaranteed during active runs and remains experimental.
+Watcher functionality is present but considered beta and may change in later releases.
 
 ## [v0.9.0] – Retreivr Rebrand Release // Music Mode and Metadata
 - Project renamed to Retreivr