Fine-grained control over toolchain components; control over SSL verification

• Fix default for toolchain --url to be $RUST_DIST_SERVER/dist. Previously this erroneously defaulted to $RUST_DIST_SERVER.

• Provide control over SSL verification via environment variables:

  • $SSL_VERIFY=false will disable verification entirely.
  • $SSL_CERT_FILE may point to a certificate file.
  • $SSL_CERT_DIR may point to directory of certificate files.

• Add toolchain all-components command. This lists all possible component names based on the manifest.

• Add toolchain --components filter. This allows processing of a subset of toolchain components.

  • Component COMP may be included via --components COMP.
  • Component COMP may be excluded via --components !COMP.

  --components may be specified multiple times. The argument to each --components switch is split on whitespace and commas to make the full list of COMP values.

• If a hash (SHA256) is provided in the manifest and thus known in advance, Romt will create the expected .sha256 files instead of downloading them. This allows Romt to work with mirrors that do not contain these .sha256 files.

• Copy manifest into canonical name in dated subdirectory if necessary. This allows Romt to work with mirrors that lack manifests in the dated directories.

  For example, Rust 1.76.0 lives in dated directory 2024-02-08. A top-level manifest of channel-rust-1.76.0.toml will be copied to the canonical location and name 2024-02-08/channel-rust-stable.toml.

• Normalize generated .sha256 format. Rust tooling pays attention to the hash value in these files but doesn't appear to care about the rest (including the text vs. binary indicator and the filename). When Romt generates a .sha256 file, it now uses * to indicate binary file hashing and uses the basename of the hashed file as the filename portion.

Lowercase prefixes, Windows archive bug fix, version upgrades

• Default to using lowercase for crate paths within archives. Most users will be unaffected by this change; however, if your crate mirror was setup using Romt prior to version 0.4.0 (release in April, 2022), you may be using mixed-case prefixes in your CRATES_ROOT without a Crate configuration file, in which case you'll need to either add the configuration file or convert your mirror to use lowercase prefixes.

• Fix Windows-specific archive file creation bug added in Romt version 0.7.0. Python's NamedTemporaryFile() allows the temporary file to be opened multiple times simultaneously on Unix, but not on Windows.

• Add continuous integration tests for Windows.

• Remove support for end-of-life Python 3.8.

• Add support for Python 3.13.

• Update Linux binaries to be built with Ubuntu 20.04 (due to the new Python 3.9 minimum supported version).

• Update Python dependency versions.

Note: no Windows executable has been provided, as Windows Defender is triggering over false-positives from the PyInstaller-generated executable.

Support crate file deletion and modification

• Support crate file deletion and modification.

  • Previously, Romt did not expect crates to be removed or modified once published; however, crates may in fact be removed from crates.io in certain circumstances, and after removal a crate may be re-published with a different hash.

  • Romt now properly handles deleted and modified crates via the newly added romt crate prune command (which is implied by the romt crate commands update, export, and import). See the "Crate file cleanup" section of README.md for information on cleaning up any modified or obsolete .crate files in an existing crates/ directory.

• Enhance the romt crate list command:

  • Support three methods of displaying crates:

    • crate@version (the new default format).
    • rel/path/to/crate-version.crate (via --show-path).
    • <SHA256SUM> *rel/path/to/crate-version.crate (via --show-hash).

  • Also display removed crates with a leading -.

• For romt crate download and romt crate verify, rename the --good-paths and --bad-paths switches to be --good-crates and --bad-crates, and use the --show-path and --show-hash switches from romt crate list to control the output format.

• Support crate filtering via --filter FILTER and --filter-file FILTER_FILE. This allows selective filtering of the crates implied by the RANGE of crates in the INDEX. Thus, for example, a single crate of a particular version may be downloaded, verified, listed, etc., e.g.:

    romt crate --start 0 --filter some_crate@1.2.3 list
  

Support older Linux distros; Add GitHub workflows for building and testing

• Speed up tests.

• Fix accidental reliance on backported security fixes in Python's tarfile module. The "data_filter" feature was added in Python 3.12, but got backported to some previous versions, making it look like it was supported since Python 3.8 (our oldest supported version). Now we probe for the feature directly to ensure it's available before we use it.

• Move build and release steps into noxfile.py.

• Add GitHub workflow for testing and quality checks.

• Add Docker-based build for Linux, based on Ubuntu 18.04. This provides executables that will run on older versions of Linux.

• Add GitHub workflow for building Romt executables.

• Change Romt's darwin alias to denote aarch64-apple-darwin now that x86_64 is no longer the primary macOS architecture.

• Note changed URL for httpx library; add some details on proxy-related environment variables for httpx.

Support "sparse" `crates.io-index` protocol

• Include poetry.lock and requirements.txt in the generated romt-x.y.z.tar.gz source archive. This allows explicit use of locked versions for all dependencies when installing from PyPI.

• Extend romt serve to support the "sparse" index protocol. This requires adjustments to the .cargo/config.toml file; see the README.rst file for details.

• Document how to use nxingx to serve the "sparse" crates.io-index protocol.

Remove extraneous artifacts from `.whl`; add build scripts to `.tar.gz.`

• Remove extraneous artifacts from the built romt-x.y.z-*.whl file. In the absence of a specified format option, these should have been present only in the source distribution file romt-x.y.z.tar.gz according to the Poetry documentation (https://python-poetry.org/docs/pyproject/#include-and-exclude). Now explicitly restrict these included files using format = "sdist". In addition, include make-exec-* and romt-wrapper.py into the source distribution file to allow building the romt executable.

Add `toolchain download --cross` feature; require Python 3.8+

• Bump minimum required interpreter version from Python 3.6 to Python 3.8.

• Add toolchain download --cross feature to allow downloading only the rust-std (Rust standard library) component of a target. This is useful for allowing cross-compilation to a given target without downloading the full native toolchain for that target.

• Update list of supported rustup targets.

• Note the use of the .toml suffix for Cargo configuration files.

• Note how to configure Cargo to use the Git command-line client for fetching the crates.io-index repository for greatly improved performance.

• Switch to Python Poetry for dependency management.

• Require git only for operations that need it.

• Tighten command-line argument parsing for shared arguments. Due to an unfortunate design aspect of Python's argparse module, "global" arguments do not work when shared between the main argument parser and subparsers. So, for example, romt --readme is accepted and correctly processed, whereas romt crate --readme is not a syntax error but the --readme switch is effectively ignored. There doesn't appear to be a clean way to work around this, so common arguments are no longer shared between the subparsers and the main parser. The two main switches (--readme and --version) must be given before any subcommand, and the remaining switches (--verbose, --quiet, --num-jobs, and --timeout) must be given after the subcommand name (e.g., romt crate --verbose).

• Allow environment variable RUSTUP_DIST_SERVER to override default value for romt toolchain --url. Allow environment variable RUSTUP_UPDATE_ROOT to override default value for romt rustup --url.

Fix case-sensitivity issues, multi-spec toolchains with `--targets=all`, add configurable download timeout

• NOTE If upgrading from older Romt, it's recommended to use the same version of Romt on the Internet-connected machine and the offline machine. See Upgrading from Romt versions before 0.4.0 in the README.rst for details.

• Add support for lowercase crate prefixes in CRATES_ROOT. This avoids problems when using a crate mirror with both case-sensitive and case-insensitive filesystems simultaneously; see #14. See README.rst for details.

• Add --timeout option to control the timeout in seconds for downloading. Change default timeout from five seconds (the default for the httpx library) to sixty seconds. A value of 0 disables the timeout altogether.

• Fix toolchain unpacking of archives created with multiple specs and --targets=all (see #17). When packing a toolchain archive, the specs and targets are specified independently, so typically each spec must use the same list of targets; but the special target all is expanded to a per-spec list of targets during packing. During unpacking, Romt had been detecting the union of all targets present in the archive and applying this set of targets to all detected specs in the archive, causing problems if one spec supported more targets than the others. Now Romt detects archives that contain all targets for all included specs and converts back to the special all target for verification and further processing.

Fix detection of toolchain targets; update `rustup` targets

• Toolchain target detection in the present of in the presence of artifacts shared across targets was incorrect, leading to spurious target detection and MissingFileError exceptions during toolchain archive unpacking; this has been fixed.

• The hard-coded list of known targets for rustup has been updated.

Fix issue #13 regarding duplicate toolchain artifact URLs.

Some distinct toolchain artifacts may share the same download URL (e.g., .../rust-docs-nightly-x86_64-unknown-linux-gnu.tar.xz is shared across several other processor variants). Before version 0.3.0, this was handled naturally by the sequential nature of the download operation, but the new asynchronous support from 0.3.0 failed to account for the possible duplication.