v0.8.0

What's Changed

Added

• New end to end file download and conversion pipeline to allow extensibility and composition of the file processing, while maintaining backwards comptability with previous Node and File classes
• Adds ResourceNode class that allows simply passing in a file path and associated metadata, and letting the file pipeline auto infer kind, file presets etc.
• Adds support for GoogleDrive download based on file URLs on Google Drive
• WebM video compression support
• Bloom format file uploads
• PDF validation in file pipeline
• Audio/video stream validation utility
• Python 3.11, 3.12, 3.13 support
• Optional Sentry integration
• Optional Slack notifications for channel uploads
• Ensure DOWNLOAD headers set a proper useragent

Changed

• The jiro CLI has been removed, as it was broadly unused, and broken.
• Dropped support for unsupported Python versions (3.7, 3.8)
• Replaced le-utils pycountry dependency with langcodes
• Default learning_activity now set based on node kind

Fixed

• Graphie processing - to make sure that non-English language data is included in Graphie files
• More robust download failure handling

Included PRs

New Features

• Download css background images by @jredrejo in #477
• Set default learning_activity based on node kind in TreeNode by @nucleogenesis in #496
• Allow uploading of bloom format files by @nikkuAg in #517
• added insufficient storage exception by @the-ivii in #567
• Add webm support to compress_video. Add tests. Raise error if webm passed to faststart. by @rtibbles in #566
• Add utility to validate stream of audio and video media files. by @rtibbles in #570
• feat: Add PDF validation to file pipeline by @rtibbles in #610
• Add optional Sentry integration by @rtibbles in #618
• Add optional Slack integration for channel upload notifications by @rtibbles in #619
• feat: Handle subtitle exceptions in SubtitleConversionHandler by @rtibbles in #624
• Google Drive handler integration by @rtibbles in #577
• Allow ContentNode class to take a uri argument for its default file by @rtibbles in #593
• Add try/finally handling for yield block, ensures that files still get checked even when caught exceptions occur. by @rtibbles in #608
• Do more robust handling of download failures. by @rtibbles in #631

Bug Fixes

• Use the right object to get the error information by @jredrejo in #469
• Use blacklisted urls in the static assets by @jredrejo in #476
• Fix name in reference to auto-thumbnail-generation argument by @jamalex in #489
• Fix placement of exception class to avoid breaking the channel manager. by @rtibbles in #583
• Revert "Allow for chefs to explicitly set a channel ID." by @rtibbles in #582
• Fix macos tests. by @rtibbles in #585
• Fix header set for the DOWNLOAD_SESSION by @GautamBytes in #596
• Fix graphie processing by @rtibbles in #604
• Fix regression introduced by overly liberal perseus image URL capture. by @rtibbles in #621
• Fix tests for latest version of yt-dlp by @rtibbles in #641
• Miscellaneous bug fixes by @rtibbles in #628

Documentation

• Update ReadTheDocs builds configuration by @MisRob in #450
• docs: update link to api token by @ThEditor in #482
• Update video_compression docs to include faststart flag by @rtibbles in #540
• Added documentation regarding the transition to WebM format (for videos) by @GautamBytes in #591
• Update installation.md to remove incorrect references to Python 3.5 by @rtibbles in #599
• Docs: Document all le_utils.constants.labels metadata fields in nodes.md by @rtibbles in #609

Testing

• Update content disposition parsing and add tests. by @rtibbles in #484
• Test under Python 3.11 & 3.12 by @rodjek in #510
• Mock yt-dlp web requests by @jredrejo in #527
• Add tests cases for predictable zip functionality. by @rtibbles in #544
• Adds more tests for the File classes to ensure behaviour ahead of refactor by @rtibbles in #571
• Add vcr for youtube test by @rtibbles in #574
• Update audio and video to same urls used in standard fixtures. by @rtibbles in #611
• Add Python 3.13 support by @rtibbles in #614

CI/CD & GitHub Actions

• Add action to send Slack notification about comments on issues by @MisRob in #479
• Feat: added update-spreadsheet action by @GarvitSinghal47 in #541
• Use notify_team_new_comment workflow action from .github repo by @rparadowski in #543
• Remove 'pull_request_review' event from the community contributions spreadsheet action by @MisRob in #548
• Trusted publishing setup by @rtibbles in #553
• Fix the action not updating some data in the contributions tracking sheet by @MisRob in #569
• feat: add community contribution labeling workflow by @iamshobhraj in #578
• Add pre-commit hooks for yml formatting and github actions linting. by @rtibbles in #586
• unassign inactive users action by @SukhvirKooner in #590
• Remove custom issue templates, turn on issue header automation, reference new contributing guidelines by @MisRob in #607
• Update manage issue header workflow events by @MisRob in #617
• Update workflow that handles contributor comments to the latest version + update make lint command to use pre-commit by @MisRob in #622
• Upgrade workflows to .github#35 by @MisRob in #643

Refactoring & Maintenance

• Drop Python 3.7 support. by @rtibbles in #459
• Do a minimum version pin for yt-dlp to allow more rapid updates by @rtibbles in #534
• Upgrade flake8 hook for Python 3.12 compatibility. by @rtibbles in #564
• Remove unused CLI to reduce complexity. by @rtibbles in #565
• Update graphie files to focus on https only downloads. by @rtibbles in #572
• Pipeline refactor by @rtibbles in #573
• Handle case where the file extension has changed from the original filename by @rtibbles in #584
• Remove reliance on le-utils based pycountry using getlang methods. by @rtibbles in #597
• Tweaks and improvements to file handling by @rtibbles in #601
• Fixes/tweak for handling compression in archive files by @rtibbles in #616
• Path handling tweaks in transfer handling by @rtibbles in #620

Dependency Upgrades

• Bump the github group with 2 updates by @dependabot[bot] in #452
• Bump urllib3 from 1.26.15 to 1.26.18 by @dependabot[bot] in #456
• Bump yt-dlp from 2023.7.6 to 2023.12.30 by @dependabot[bot] in #458
• Bump sphinx from 2.4.4 to 7.2.6 by @dependabot[bot] in #453
• Bump the github group with 1 update by @dependabot[bot] in https://github.com/learningequalit...

v0.7.3

What's Changed

• Update requirements configuration in setup.py. by @rtibbles in #425
• Bump cachecontrol from 0.12.0 to 0.12.11 by @dependabot in #428
• Bump pytest-env from 0.6.2 to 0.8.1 by @dependabot in #399
• Bump pdf2image from 1.11.0 to 1.16.3 by @dependabot in #426
• Fixed filetype for DocumentFile by @lsolesen in #432
• Add StudioContentNode class to allow scripted curation of existing content from Studio by @jamalex in #412

New Contributors

• @lsolesen made their first contribution in #432

Full Changelog: v0.7.2...v0.7.3

v0.7.2

What's Changed

• Fix youtube downloads by upgrading youtube dl by @rtibbles in #417
• Fix encoding when persisting trees to JSON by adding encoding on reading json file by @AtKristijan in #415

Dependency updates:

• Added encoding on reading json file by @AtKristijan in #415
• Bump actions/checkout from 2 to 3 by @dependabot in #421
• Bump actions/cache from 2 to 3 by @dependabot in #422
• Bump requests-cache from 0.4.13 to 1.0.1 by @dependabot in #413
• Bump actions/setup-python from 2 to 4 by @dependabot in #420
• Bump pre-commit/action from 2.0.0 to 3.0.0 by @dependabot in #423

Full Changelog: v0.7.1...v0.7.2

v0.7.1

What's Changed

• Fix argument passed to do_file_upload - fixes issue with 400s on file uploads by @rtibbles in #402
• Compression updates to allow compression of audio files, blanket compression of video files, and compression of media files inside zip/H5P by @rtibbles in #409

Full Changelog: v0.7.0...v0.7.1

v0.7.0

What's Changed

Added

• Add YouTube caching util by @WenyuZhang1992 in #278
• Improve exercise validation for CSV workflow by @kollivier in #282
• Add support for persistent ricecooker settings. by @kollivier in #288
• Allow for chefs to explicitly set a channel ID. by @kollivier in #285
• Make it possible for HTML zips to specify which page to load in cases… by @kollivier in #286
• Add parallel download function. by @kollivier in #289
• Save metadata to csv by @richard-dinh in #290
• Add YouTubeSushiChef base class for YouTube chefs by @kollivier in #294
• Add command line tool for creating and running sushi chefs. by @kollivier in #287
• Support creation of page-relative links in ArchiveDownloader, and make it default by @kollivier in #306
• Custom navigation update by @rtibbles in #304
• sort_topic_nodes added to TreeNode by @richard-dinh in #307
• Adds webm formats and tests by @sairina in #305
• Add MacOS and Windows to the test runner by @rtibbles in #319
• Allow manual presets for files by @AtKristijan in #344
• Direct GCS Upload by @rtibbles in #339
• Add PracticeQuizNode. by @rtibbles in #349
• extract video duration. by @AtKristijan in #352
• Make strict mode optional to allow partial uploads by @rtibbles in #373
• Get ext from request content disposition header by @AtKristijan in #372
• Allow adding metadata labels. by @AtKristijan in #370
• Add file duration at upload by @rtibbles in #379
• Add support for not processing files that already exist on remote by @rtibbles in #381
• Add support for Python 3.10 by @rtibbles in #383

Fixed

• Don't point to the api.studio domain for (p)review. by @kollivier in #283
• Handle the case where m / n are not set. by @kollivier in #284
• Improve cache checking to avoid copying unchanged files. by @kollivier in #292
• Fix multithreading and memory issue with using DOWNLOAD_SESSION… by @kollivier in #293
• Fix issue where we unset domain when we shouldn't… by @kollivier in #295
• Fix issues with relative CSS url() detection… by @kollivier in #296
• Fix for issue with rewriting relative urls when using get_archive_filename by @kollivier in #297
• Docs: Remove mentions of deprecated flags by @bkarnow in #302
• Minor fixes to PDF utilities documentation by @bkarnow in #310
• Only guess encoding when response encoding doesn't exist by @AtKristijan in #346
• Instead of checking download path to check page url in archive_page by @AtKristijan in #345
• Fix Pre commit for ricecooker by @AtKristijan in #351
• Fix file metadata for questions to ensure they can be uploaded properly. by @rtibbles in #353
• fix: tag length validation by @vkWeb in #355
• basetemp remove by @AtKristijan in #359
• revert by @AtKristijan in #365
• Use updated youtube-dl fork by @AtKristijan in #364
• Fix pytest and tox by @AtKristijan in #362
• Do processing of all image files to ensure compliance with requirements. by @rtibbles in #369
• Coerce all file errors to strings to let them be pickled. by @rtibbles in #374
• fixing problem with h5p_thumbnail by @AtKristijan in #377
• Update how we handle test media files for compression utils. by @rtibbles in #386
• Fix upload check by doing replacement properly and using download session. by @rtibbles in #387
• Bump pycountry from 17.5.14 to 22.3.5 by @dependabot in #391
• Add workaround for bug in file upload endpoint. by @rtibbles in #393
• Default values for some metadata on ContentNode set to list() by @nucleogenesis in #394

Changed

• Update CI to Github Actions by @rtibbles in #316
• Update windows ffmpeg download instructions. by @rtibbles in #317
• Test cleanup by @rtibbles in #318
• Vendor pressurecooker by @rtibbles in #340
• autopep8, autoflake8, manualjake8 linting by @nucleogenesis in #361

New Contributors

• @WenyuZhang1992 made their first contribution in #278
• @richard-dinh made their first contribution in #290
• @bkarnow made their first contribution in #302
• @rtibbles made their first contribution in #304
• @sairina made their first contribution in #305
• @AtKristijan made their first contribution in #344
• @vkWeb made their first contribution in #355
• @dependabot made their first contribution in #391

Full Changelog: v0.6.45...v0.7.0

Better docs, corrections script, and `libstudio` utils

• Updated docs build scripts to make ricecooker docs available on read the docs
• Added corrections command line script for making bulk edits to content metadata
• Added StudioApi client to support CRUD (created, read, update, delete) Studio actions
• Added pdf-splitting helper methods (see ricecooker/utils/pdf.py)

ePub, tagging, and bugfixes

0.6.24 (2018-11-09)

• Updated le-utils and pressurcooker dependencies to latest version
• Added support for ePub files (EPubFiles can be added of DocumentNodes)
• Added tag support
• Changed default value for STUDIO_URL to api.studio.learningequality.org
• Added aggregator and provider fields for content nodes
• Various bugfixes to image processing in exercises
• Added is_youtube_subtitle_file_supported_language helper function to support importing youtube subs
• Added srt2vtt subtitles conversion
• Added static assets downloader helper method in utils.downloader.download_static_assets
• Added LineCook chef functions to --generate CSV from directory structure
• Fixed the always randomize=True bug
• Docs: general content node metadata guidelines
• Docs: video compression instructions and helper scripts convertvideo.bat and convertvideo.sh

Node roles and new docs

• Added support for role attribute on ConentNodes (currently coach || learner)
• Update pressurecooker dependency (to catch compression errors)
• Docs improvements, see https://github.com/learningequality/ricecooker/tree/master/docs

Spring cleaning and bugfixes

General update and some new features:
https://github.com/learningequality/ricecooker/blob/master/docs/history.rst#0615-2018-03-06

v0.3.14: Merge pull request #41 from jayoshih/pressure-on

Pressurecooker Integration