feat: compress repo runtime snapshots

Author: yaochao

README.en.md

@@ -7,7 +7,7 @@ This repository does one specific thing: on a modern host, it builds the two run
 The repository no longer stores third-party runtime images. The repository now includes self-built runtime image snapshots under version control:
 
 - `images/bootimage-0.12-hd`
-- `images/hdc-0.12.img`
+- `images/hdc-0.12.img.xz`
 
 The same source-build workflow also produces local working images:
 
@@ -23,7 +23,7 @@ The build and runtime flow is:
 - compile the kernel boot image
 - compile the repo-owned minimal userland programs `/bin/sh` and `/bin/ls`
 - build a Minix v1 root filesystem from repo manifests
-- generate the repo-bundled images `images/bootimage-0.12-hd` and `images/hdc-0.12.img`
+- generate the repo-bundled images `images/bootimage-0.12-hd` and `images/hdc-0.12.img.xz`
 - boot QEMU
 - reach `[/usr/root]#`
 - run `ls`
@@ -113,7 +113,7 @@ Windows CMD:
 scripts\run.cmd
 ```
 
-This entrypoint uses the committed images in `images/` directly and does not rebuild first. On macOS / Ubuntu it keeps the current terminal-based interactive flow; on Windows it already uses a visible GUI window.
+This entrypoint uses the committed snapshots in `images/` directly and does not rebuild first. The hard disk image is automatically unpacked to `out/repo-images/hdc-0.12.img` before launch. On macOS / Ubuntu it keeps the current terminal-based interactive flow; on Windows it already uses a visible GUI window.
 
 ### 4. Open A Visible QEMU Window And Interact Manually
 
@@ -161,7 +161,7 @@ Windows CMD:
 scripts\build-and-run.cmd
 ```
 
-This entrypoint forces a rebuild, syncs the new images into `images/`, and then starts QEMU.
+This entrypoint forces a rebuild, syncs the new images into `images/`, stores the hard disk image as the compressed snapshot `images/hdc-0.12.img.xz`, and then starts QEMU.
 
 If you want the flow to start from compilation and still end in a visible interactive QEMU window, run:
 
@@ -299,7 +299,8 @@ Important generated artifacts:
 - `rebuild/out/images/bootimage-0.12-hd`
 - `rebuild/out/images/hdc-0.12.img`
 - `images/bootimage-0.12-hd`
-- `images/hdc-0.12.img`
+- `images/hdc-0.12.img.xz`
+- `out/repo-images/hdc-0.12.img`
 - `out/verify/screen.txt`
 - `out/verify-userland/screen.txt`
 - `out/run/boot.img`
@@ -316,7 +317,7 @@ The `rebuild/` directory owns the full source-to-image pipeline:
 6. create directories, device nodes, and boot files from `rebuild/rootfs/manifest/`
 7. build a Minix v1 root filesystem that Linux 0.12 can mount
 8. assemble `hdc-0.12.img`
-9. sync new images into the repo-managed `images/` directory when requested
+9. sync the new boot image and compressed hard disk snapshot into the repo-managed `images/` directory when requested
 10. boot QEMU, scrape VGA text, and inject keys to complete verification
 
 This pipeline intentionally builds only the smallest system required by the repo. It does not try to recreate a full historical Linux 0.12 distribution.
@@ -340,7 +341,7 @@ The current standalone userland binaries are:
 - `scripts/`
   host-specific entry scripts
 - `images/`
-  committed snapshots of the self-built runtime images
+  committed snapshots of the self-built runtime images, with the hard disk stored in compressed form
 - `rebuild/driver.py`
   source build, runtime, and verification entrypoint
 - `rebuild/container/build_images.sh`
@@ -363,8 +364,8 @@ The current standalone userland binaries are:
 ## Runtime Notes
 
 - The boot image is shorter than 1.44MB, so the driver pads it into a full floppy image before launch
-- `scripts/run.*` uses the committed images in `images/` by default
-- `scripts/run-window.*` uses the committed images in `images/` and opens a visible QEMU window
+- `scripts/run.*` uses the committed snapshots in `images/` by default and unpacks the hard disk image into `out/repo-images/`
+- `scripts/run-window.*` uses the committed snapshots in `images/`, unpacks the hard disk image into `out/repo-images/`, and opens a visible QEMU window
 - `scripts/build-and-run.*` rebuilds from source and refreshes `images/`
 - `scripts/build-and-run-window.*` rebuilds from source, refreshes `images/`, and opens a visible QEMU window
 - QEMU always starts with `-snapshot`, so repeated runs do not mutate `rebuild/out/images/hdc-0.12.img`

README.md

@@ -7,7 +7,7 @@
 仓库当前不再保存第三方运行时镜像。仓库里提交的是本项目自己编译出来的镜像快照：
 
 - `images/bootimage-0.12-hd`
-- `images/hdc-0.12.img`
+- `images/hdc-0.12.img.xz`
 
 同一套源码构建链还会在本地生成工作镜像：
 
@@ -23,7 +23,7 @@
 - 编译内核启动镜像
 - 编译仓库自带的最小用户态程序 `/bin/sh` 和 `/bin/ls`
 - 按仓库清单生成 Minix v1 根文件系统
-- 生成仓库内置镜像 `images/bootimage-0.12-hd` 和 `images/hdc-0.12.img`
+- 生成仓库内置镜像 `images/bootimage-0.12-hd` 和 `images/hdc-0.12.img.xz`
 - 启动 QEMU
 - 进入 `[/usr/root]#`
 - 执行 `ls`
@@ -113,7 +113,7 @@ Windows CMD：
 scripts\run.cmd
 ```
 
-这条入口直接使用仓库里的 `images/` 镜像，不会先重编源码。macOS / Ubuntu 下它保留当前的终端交互方式；Windows 下本来就是图形窗口。
+这条入口直接使用仓库里的 `images/` 快照，不会先重编源码。系统硬盘镜像会先自动解包到 `out/repo-images/hdc-0.12.img`，然后再启动。macOS / Ubuntu 下它保留当前的终端交互方式；Windows 下本来就是图形窗口。
 
 ### 4. 弹出可见的 QEMU 窗口并手动操作
 
@@ -161,7 +161,7 @@ Windows CMD：
 scripts\build-and-run.cmd
 ```
 
-这条入口会强制重新编译，完成后把新的镜像同步到 `images/`，然后启动 QEMU。
+这条入口会强制重新编译，完成后把新的镜像同步到 `images/`，其中硬盘镜像会以压缩快照形式写成 `images/hdc-0.12.img.xz`，然后启动 QEMU。
 
 如果你既要“从编译开始”，又要最后看到可交互的 QEMU 窗口，运行：
 
@@ -299,7 +299,8 @@ python3 rebuild/driver.py run
 - `rebuild/out/images/bootimage-0.12-hd`
 - `rebuild/out/images/hdc-0.12.img`
 - `images/bootimage-0.12-hd`
-- `images/hdc-0.12.img`
+- `images/hdc-0.12.img.xz`
+- `out/repo-images/hdc-0.12.img`
 - `out/verify/screen.txt`
 - `out/verify-userland/screen.txt`
 - `out/run/boot.img`
@@ -316,7 +317,7 @@ python3 rebuild/driver.py run
 6. 根据 `rebuild/rootfs/manifest/` 创建目录、设备节点和启动脚本
 7. 生成 Linux 0.12 可挂载的 Minix v1 根文件系统
 8. 组装成 `hdc-0.12.img`
-9. 按需把新镜像同步到仓库的 `images/`
+9. 按需把新的启动镜像和压缩后的系统镜像同步到仓库的 `images/`
 10. 启动 QEMU，抓取 VGA 文本，并自动向 guest 发送按键完成验证
 
 这条链路当前只实现“最小可运行系统”，不尝试复刻一个完整的历史 Linux 0.12 发行版。
@@ -340,7 +341,7 @@ python3 rebuild/driver.py run
 - `scripts/`
   不同宿主机的入口脚本
 - `images/`
-  提交到仓库中的自编译运行镜像快照
+  提交到仓库中的自编译运行镜像快照，其中系统镜像以压缩形式保存
 - `rebuild/driver.py`
   源码构建、运行和验证入口
 - `rebuild/container/build_images.sh`
@@ -363,8 +364,8 @@ python3 rebuild/driver.py run
 ## 运行说明
 
 - 启动镜像本身不足 1.44MB，驱动会在启动前补齐成标准软盘镜像
-- `scripts/run.*` 默认直接使用仓库里的 `images/`
-- `scripts/run-window.*` 默认直接使用仓库里的 `images/`，并弹出可见的 QEMU 窗口
+- `scripts/run.*` 默认直接使用仓库里的 `images/` 快照，并在 `out/repo-images/` 里自动解包系统镜像
+- `scripts/run-window.*` 默认直接使用仓库里的 `images/` 快照，并在 `out/repo-images/` 里自动解包系统镜像后弹出可见的 QEMU 窗口
 - `scripts/build-and-run.*` 会重编源码并刷新 `images/`
 - `scripts/build-and-run-window.*` 会重编源码、刷新 `images/`，然后弹出可见的 QEMU 窗口
 - QEMU 始终以 `-snapshot` 启动，所以重复运行不会改写 `rebuild/out/images/hdc-0.12.img`

images/README.en.md

@@ -2,17 +2,17 @@
 
 [中文 README](./README.md)
 
-This directory stores the repo-managed runtime image snapshots. They are not third-party downloads. They are built by this repository's own `rebuild/` pipeline and then synced here.
+This directory stores the repo-managed runtime image snapshots. They are not third-party downloads. They are built by this repository's own `rebuild/` pipeline and then synced here. To keep the repository smaller, the hard disk image is stored as a compressed snapshot.
 
 ## Current Files
 
 - `bootimage-0.12-hd`
   Linux 0.12 boot image
-- `hdc-0.12.img`
-  Linux 0.12 minimal system hard disk image
+- `hdc-0.12.img.xz`
+  compressed snapshot of the Linux 0.12 minimal system hard disk image
 
 ## Usage
 
-- `scripts/run.*` boots directly from the images here
-- `scripts/run-window.*` boots directly from the images here and opens a visible window
-- `scripts/build-and-run.*` and `scripts/build-and-run-window.*` rebuild first, then refresh the images here
+- `scripts/run.*` boots from the snapshots here and unpacks the hard disk image into `out/repo-images/`
+- `scripts/run-window.*` boots from the snapshots here, unpacks the hard disk image into `out/repo-images/`, and opens a visible window
+- `scripts/build-and-run.*` and `scripts/build-and-run-window.*` rebuild first, then refresh the snapshots here

images/README.md

@@ -2,17 +2,17 @@
 
 [English README](./README.en.md)
 
-这个目录保存受版本控制的运行镜像快照。它们不是第三方下载镜像，而是本仓库通过 `rebuild/` 流程自行构建出来并同步到这里的结果。
+这个目录保存受版本控制的运行镜像快照。它们不是第三方下载镜像，而是本仓库通过 `rebuild/` 流程自行构建出来并同步到这里的结果。为了减小仓库体积，系统硬盘镜像以压缩快照形式保存。
 
 ## 当前文件
 
 - `bootimage-0.12-hd`
   Linux 0.12 启动镜像
-- `hdc-0.12.img`
-  Linux 0.12 最小系统硬盘镜像
+- `hdc-0.12.img.xz`
+  Linux 0.12 最小系统硬盘镜像的压缩快照
 
 ## 使用方式
 
-- `scripts/run.*` 直接使用这里的镜像启动
-- `scripts/run-window.*` 直接使用这里的镜像并弹出可见窗口
-- `scripts/build-and-run.*` 和 `scripts/build-and-run-window.*` 会先重编，再刷新这里的镜像
+- `scripts/run.*` 会直接使用这里的快照，并把系统镜像自动解包到 `out/repo-images/`
+- `scripts/run-window.*` 会直接使用这里的快照，并把系统镜像自动解包到 `out/repo-images/` 后弹出可见窗口
+- `scripts/build-and-run.*` 和 `scripts/build-and-run-window.*` 会先重编，再刷新这里的镜像快照

images/hdc-0.12.img

@@ -9,7 +9,7 @@
 - build `rebuild/out/images/bootimage-0.12-hd`
 - build `rebuild/out/images/hdc-0.12.img`
 - use `driver.py` to orchestrate Docker and QEMU for build, boot, and verification
-- sync built images into the repo-level `images/` directory when requested
+- sync the built boot image and compressed hard disk snapshot into the repo-level `images/` directory when requested
 
 ## Main Contents
 
@@ -41,4 +41,4 @@ python3 rebuild/driver.py run-repo-images-window
 
 - this directory owns source build and image assembly
 - runtime output under `rebuild/out/` is generated content
-- repo-managed image snapshots live in the top-level `images/` directory
+- repo-managed image snapshots live in the top-level `images/` directory, with the hard disk stored in compressed form

images/hdc-0.12.img.xz

@@ -9,7 +9,7 @@
 - 构建 `rebuild/out/images/bootimage-0.12-hd`
 - 构建 `rebuild/out/images/hdc-0.12.img`
 - 通过 `driver.py` 调用 Docker 和 QEMU 完成构建、启动与验证
-- 在需要时把构建出来的镜像同步到仓库根目录的 `images/`
+- 在需要时把构建出来的启动镜像和压缩后的系统镜像同步到仓库根目录的 `images/`
 
 ## 主要内容
 
@@ -41,4 +41,4 @@ python3 rebuild/driver.py run-repo-images-window
 
 - 这里只管理源码构建和镜像装配流程
 - 运行时输出在 `rebuild/out/`，属于生成产物
-- 受版本控制的镜像快照在仓库根目录 `images/`
+- 受版本控制的镜像快照在仓库根目录 `images/`，其中系统镜像以压缩形式保存

rebuild/README.en.md

@@ -1,6 +1,7 @@
 from __future__ import annotations
 
 import argparse
+import lzma
 import os
 import shutil
 import subprocess
@@ -24,14 +25,17 @@ class BuildPaths:
     boot_image: Path
     hard_disk_image: Path
     repo_boot_image: Path
-    repo_hard_disk_image: Path
+    repo_hard_disk_image_archive: Path
+    repo_runtime_dir: Path
+    repo_runtime_hard_disk_image: Path
 
     @classmethod
     def from_root(cls, root: Path) -> "BuildPaths":
         rebuild_dir = root / "rebuild"
         out_dir = rebuild_dir / "out"
         images_dir = out_dir / "images"
         repo_images_dir = root / "images"
+        repo_runtime_dir = root / "out" / "repo-images"
         return cls(
             root=root,
             repo_images_dir=repo_images_dir,
@@ -43,7 +47,9 @@ def from_root(cls, root: Path) -> "BuildPaths":
             boot_image=images_dir / "bootimage-0.12-hd",
             hard_disk_image=images_dir / "hdc-0.12.img",
             repo_boot_image=repo_images_dir / "bootimage-0.12-hd",
-            repo_hard_disk_image=repo_images_dir / "hdc-0.12.img",
+            repo_hard_disk_image_archive=repo_images_dir / "hdc-0.12.img.xz",
+            repo_runtime_dir=repo_runtime_dir,
+            repo_runtime_hard_disk_image=repo_runtime_dir / "hdc-0.12.img",
         )
 
 
@@ -91,7 +97,7 @@ def verify_environment(paths: BuildPaths, source: str = "rebuild") -> dict[str,
         return env
     if source == "repo":
         env["LINUX012_BOOT_SOURCE_IMAGE"] = str(paths.repo_boot_image)
-        env["LINUX012_HARD_DISK_IMAGE"] = str(paths.repo_hard_disk_image)
+        env["LINUX012_HARD_DISK_IMAGE"] = str(paths.repo_runtime_hard_disk_image)
         return env
     raise ValueError(f"Unsupported runtime image source: {source}")
 
@@ -153,7 +159,7 @@ def ensure_rebuilt_images(paths: BuildPaths) -> int:
 
 
 def require_repo_images(paths: BuildPaths) -> bool:
-    required = [paths.repo_boot_image, paths.repo_hard_disk_image]
+    required = [paths.repo_boot_image, paths.repo_hard_disk_image_archive]
     missing = [path for path in required if not path.exists() or path.stat().st_size == 0]
     if not missing:
         return True
@@ -164,12 +170,44 @@ def require_repo_images(paths: BuildPaths) -> bool:
     return False
 
 
+def compress_image_snapshot(source: Path, target: Path) -> None:
+    target.parent.mkdir(parents=True, exist_ok=True)
+    temporary = target.with_name(f"{target.name}.tmp")
+    with source.open("rb") as source_handle, lzma.open(temporary, "wb", preset=9 | lzma.PRESET_EXTREME) as target_handle:
+        shutil.copyfileobj(source_handle, target_handle)
+    temporary.replace(target)
+
+
+def extract_image_snapshot(source: Path, target: Path) -> None:
+    target.parent.mkdir(parents=True, exist_ok=True)
+    temporary = target.with_name(f"{target.name}.tmp")
+    with lzma.open(source, "rb") as source_handle, temporary.open("wb") as target_handle:
+        shutil.copyfileobj(source_handle, target_handle)
+    temporary.replace(target)
+    os.utime(target, ns=(source.stat().st_mtime_ns, source.stat().st_mtime_ns))
+
+
 def sync_repo_images(paths: BuildPaths) -> int:
     if not require_rebuilt_images(paths):
         return 1
     paths.repo_images_dir.mkdir(parents=True, exist_ok=True)
     shutil.copy2(paths.boot_image, paths.repo_boot_image)
-    shutil.copy2(paths.hard_disk_image, paths.repo_hard_disk_image)
+    compress_image_snapshot(paths.hard_disk_image, paths.repo_hard_disk_image_archive)
+    legacy_image = paths.repo_images_dir / "hdc-0.12.img"
+    if legacy_image.exists():
+        legacy_image.unlink()
+    return 0
+
+
+def ensure_repo_runtime_images(paths: BuildPaths) -> int:
+    if not require_repo_images(paths):
+        return 1
+    archive = paths.repo_hard_disk_image_archive
+    runtime_image = paths.repo_runtime_hard_disk_image
+    if runtime_image.exists() and runtime_image.stat().st_size > 0:
+        if runtime_image.stat().st_mtime_ns >= archive.stat().st_mtime_ns:
+            return 0
+    extract_image_snapshot(archive, runtime_image)
     return 0
 
 
@@ -194,7 +232,7 @@ def run_runtime(paths: BuildPaths, mode: str) -> int:
 
 
 def run_repo_runtime(paths: BuildPaths, mode: str = "run") -> int:
-    if not require_repo_images(paths):
+    if ensure_repo_runtime_images(paths) != 0:
         return 1
     return run_command(
         [sys.executable, str(paths.root / "tools" / "qemu_driver.py"), mode],

rebuild/README.md

@@ -9,9 +9,9 @@ This directory stores host-facing one-command entry scripts. Each platform has i
 - `bootstrap-host.*`
   check host dependencies such as Python, QEMU, and Docker
 - `run.*`
-  boot directly from the repo-managed images
+  boot directly from the repo-managed snapshots and unpack the hard disk image automatically
 - `run-window.*`
-  boot directly from the repo-managed images and open a visible QEMU window
+  boot directly from the repo-managed snapshots, unpack the hard disk image automatically, and open a visible QEMU window
 - `build-and-run.*`
   rebuild first, then boot
 - `build-and-run-window.*`