Repository clone size is unnecessarily large due to deleted binary in history and untracked large files

[KeitaW]

Problem

The .git directory weighs 177 MB despite roughly 72 MB of working tree content, making fresh clones significantly slower than necessary. A single 164 MB pack file containing 12,525 objects accounts for nearly all of this overhead, and git count-objects -vH reports 5,599 non-delta objects, indicating a substantial proportion of binary or incompressible content.

Investigation methodology

The analysis combined several approaches to identify where the bloat originates:

• Measured the .git directory size with du -sh .git and inspected ls -lh .git/objects/pack/ to quantify the overhead at 177 MB total, dominated by a single 164 MB pack file.
• Ran git count-objects -vH for git's own size accounting, confirming 5,599 non-delta objects — a high count that points to many binary files.
• Enumerated the largest blobs across all history using git rev-list --objects --all | git cat-file --batch-check to surface objects that persist in the pack regardless of whether they still exist in the current working tree.
• Scanned the working tree for large files by extension (GIF, ZIP, PNG, tar.bz2, CSV, JSON) using find with size sorting to identify current binary bloat.
• Checked git log --diff-filter=D --summary to identify large files that were committed and later deleted but still consume space in the object database.
• Verified .gitattributes contents and checked git lfs ls-files — Git LFS is not configured.
• Measured per-directory sizes with du -sh */ to understand which areas of the repo contribute most.

Root cause analysis

The following table lists the sources of bloat in descending order of impact:

Source	Size	Notes
Deleted miniconda.sh at 3.test_cases/9.nemo-multimodal/nemo_configs/miniconda.sh	95.8 MB	Added in commit d0227b2, deleted in commit 32b3313 (Oct 2023). Still stored in the git object database and downloaded by every clone. This single file accounts for ~54% of the .git directory size.
Demo GIF files (automate-smhp-demo.gif, automate-smhp-eks-demo.gif)	~30 MB	Two animated GIFs used for documentation.
Lambda function ZIP (grafana-service-token-lambda-function.zip)	15 MB	Binary archive checked into the tree.
PNG screenshots across 0.docs/ and other directories	~10 MB	Dozens of image files spread across multiple directories.
Other binaries (hwloc-2.9.2-h2bc3f7f_0.tar.bz2, slurm-esm1nv-train-102.out, etc.)	~7 MB	Miscellaneous archives and log files.

Proposed next steps

The most impactful remediation is purging the deleted miniconda.sh from history, which alone would reclaim roughly 96 MB. Beyond that, migrating current large binaries to Git LFS and adding guardrails would prevent recurrence.

Specifically:

1. Purge miniconda.sh from history using git filter-repo --path 3.test_cases/9.nemo-multimodal/nemo_configs/miniconda.sh --invert-paths.
2. Migrate large binaries (GIFs, ZIPs, archives) to Git LFS via git lfs migrate import.
3. Add .gitattributes rules to enforce LFS tracking for *.gif, *.zip, *.tar.bz2, *.png, and similar extensions.
4. The expected result is roughly a 60% reduction in clone size (from 177 MB down to approximately 60–70 MB).

Rollout considerations

Because history rewriting is a breaking change for existing clones and forks, the following precautions are warranted:

• Announce to contributors before the history rewrite so they can prepare.
• Combine the filter-repo purge and LFS migration into a single coordinated force push to minimize disruption.
• Update contributing guidelines to mention git lfs install as a prerequisite for development.
• Provide re-sync instructions for existing forks (e.g., fresh clone or git fetch --refetch).

[KeitaW]

Contributor Instructions: Post-Cleanup Steps

After the repository cleanup is complete, all contributors will need to take action. The cleanup involves a history rewrite (purging a 96 MB deleted binary) and a Git LFS migration (moving large files to LFS), both of which result in a force push that changes the entire commit history.

---

1. Anyone with a local clone (no uncommitted work)

The simplest path — delete and re-clone:

rm -rf awsome-distributed-training
git clone https://github.com/aws-samples/awsome-distributed-training.git

2. Anyone with a local clone and uncommitted/local work

# Save your work first
git stash   # or commit to a local branch

# Re-fetch with rewritten history
git fetch --refetch origin
git checkout main
git reset --hard origin/main

# If you had a local branch with in-progress work:
git checkout my-branch
git rebase --onto origin/main <old-base-commit> my-branch
# You will likely need to resolve conflicts

3. Anyone with a fork on GitHub

Forks don't auto-sync after a force push. Contributors must:

# In their fork's local clone
git remote set-url origin https://github.com/<their-fork>/awsome-distributed-training.git
git remote add upstream https://github.com/aws-samples/awsome-distributed-training.git

# Fetch the rewritten history
git fetch upstream

# Hard reset their main to match upstream
git checkout main
git reset --hard upstream/main
git push origin main --force

4. Anyone with open Pull Requests

Open PRs will become unmergeable because their base commits no longer exist. Contributors must:

1. Update their fork (step 3 above)
2. Rebase their PR branch onto the new main:

   git checkout my-pr-branch
   git rebase --onto upstream/main <old-base-commit> my-pr-branch
   git push origin my-pr-branch --force

3. Or, more practically: create a fresh branch from new main, cherry-pick or re-apply their changes, and open a new PR.

5. New prerequisite: Git LFS

All contributors going forward need Git LFS installed:

# Install Git LFS (one-time)
git lfs install

# This is now required before cloning, otherwise large
# files will appear as pointer files instead of actual content

6. New rules for adding files

Contributors must follow the new .gitattributes rules:

• Never commit raw *.gif, *.zip, *.tar.bz2, *.png, or other large binaries directly — they are now tracked by LFS automatically via .gitattributes.
• If adding a new binary file type, add a corresponding LFS tracking rule.

---

Summary: The easiest path for most contributors is to delete and re-clone. If you have in-progress work or open PRs, follow the detailed steps above. Going forward, git lfs install is a prerequisite for development.

We apologize for the disruption — this is a one-time operation that reduces clone size from ~177 MB to ~65 MB.

[KeitaW]

Repository Cleanup Complete

The repository cleanup has been successfully completed. Here are the results:

Before/After Metrics

Metric	Before	After
.git size (clone)	166 MB	26 MB
Clone time	~5.0s	~4.2s
Largest blob in history	95.8 MB (miniconda.sh)	Removed
LFS-tracked types	none	*.gif, *.zip, *.tar.bz2, *.tar.gz

What was done

1. Purged 3 deleted files from history using git-filter-repo:

   • 3.test_cases/9.nemo-multimodal/nemo_configs/miniconda.sh (95.8 MB)
   • 4.validation_and_observability/5.nsight/slurm-esm1nv-train-102.out (~5.4 MB)
   • 3.test_cases/17.SM-modelparallelv2/bin/hwloc-2.9.2-h2bc3f7f_0.tar.bz2 (~1.2 MB)

2. Migrated large binaries to Git LFS (*.gif, *.zip, *.tar.bz2, *.tar.gz) across all branches and tags

3. Updated repository files:

   • .gitattributes — Added Git LFS tracking rules
   • CONTRIBUTING.md — Added Git LFS prerequisite section for contributors

Impact on contributors

• Existing clones: Contributors will need to re-clone or run git fetch --refetch origin && git reset --hard origin/main
• New clones: git lfs install is required before cloning (one-time setup)
• Open PRs: Will need to be rebased onto the new history

Backup

A full mirror backup was created before the cleanup and is retained for recovery if needed.

[KeitaW]

Repository Cleanup Completed — PR Recovery In Progress

The repository history cleanup described in this issue has been completed. The following changes were applied:

1. Purged 3 large files from history using git filter-repo:
   • 3.test_cases/9.nemo-multimodal/nemo_configs/miniconda.sh (95.8 MB)
   • 1.architectures/7.sagemaker-hyperpod-eks/automate-smhp-demo.gif (16.1 MB)
   • 1.architectures/7.sagemaker-hyperpod-eks/automate-smhp-eks-demo.gif (14.3 MB)
2. Migrated large binaries to Git LFS (.gif, .zip, .tar.bz2, .png, .csv files)
3. Updated .gitattributes to enforce LFS tracking for these file types going forward
4. Force-pushed rewritten history to all branches and tags

Impact on Open PRs

The force push caused GitHub to auto-close 13 previously open PRs because their base commits no longer exist in the rewritten history. These fall into two categories:

---

Same-repo PRs (7) — being reopened/recreated now

These branches were also rewritten during the cleanup, so they already have correct commits relative to the new main. We are reopening or recreating these PRs:

PR	Branch	Author	Title
#950	hyperpod-slurm-cpu-memory-metrics	@KeitaW	Add observability support for HyperPod Slurm login nodes
#947	tf-hpio-update	@bluecrayon52	Terraform Updates for HyperPod Inference Operator v2 Official EKS addon
#911	removing_AWS_OFI_NCCL_VERSION	@pbelevich	Remove AWS_OFI_NCCL_VERSION
#895	fix/pr-885-efa-node-exporter-improvements	@paragao	Syntax improvements for EFA node exporter
#864	nanoVLM	@allela-roy	Adding nanoVLM sample
#838	gb200-smhp-slurm	@amanshanbhag	Updating CF stack for GB200 local zone deployments
#747	adding-tf-eks	@KeitaW	Adding terraform EKS reference architecture

No action required from these authors — the PRs are being restored with the same content.

---

Cross-repo (fork) PRs (6) — require action from fork owners

These PRs came from forks. The fork branches still reference the old (pre-rewrite) history, so they cannot be reopened automatically. Fork owners need to rebase their branches onto the new main:

PR	Branch	Owner	Title
#930	feature/hyperpod-training-plan	@newabdosheham	Add optional SageMaker Training Plan support
#871	update-eni-sg	@paragao	Helper tool to add new security groups to FSxL ENIs
#808	main	@emeraldbay	Add HyperpodTrainingOperatorServiceRole in CF template
#807	main	@cszhz	Delete users script in hyperpod
#804	feature/slinky-slurm-hyperpod-eks	@bdaqiq01	Feature/slinky slurm hyperpod eks
#708	scape-up-down-instance-group-nodes-tool	@paragao	Scale up-down nodes tool

Instructions for fork PR owners

To update your fork and reopen your PR:

# 1. Update your fork's main branch
cd your-fork
git fetch upstream          # upstream = aws-samples/awsome-distributed-training
git checkout main
git reset --hard upstream/main
git push --force origin main

# 2. Rebase your feature branch
git checkout your-feature-branch
git rebase main
# Resolve any conflicts if needed
git push --force origin your-feature-branch

# 3. Open a new PR from your fork targeting aws-samples/awsome-distributed-training:main

If you have trouble rebasing, you can also cherry-pick your changes onto a fresh branch from the new main.

---

Comments have been posted on each of the 6 fork PRs with individual notifications and instructions.