Mismatch between gut and git

[snomos]

gut pull -a reports some dirty repos for me:

+---------------------------------------------------+
| Organisation       #repos  Updated  Failed  Dirty |
+===================================================+
| apertium              599        0       0      0 |
| borealium               3        0       0      0 |
| divvun                155        1       0      1 |
| divvungiellatekno      33        0       0      1 |
| giellalt              558        5       0      0 |
| giellatekno            41        0       0      1 |
| pahkat                  6        0       0      0 |
| skuvla                  2        0       0      0 |
|                                                   |
| TOTAL                1397        6       0      3 |
+---------------------------------------------------+

In the divvungiellatekno org, the repo template-keynote-UiT is reported as dirty. And gut status -o divvungiellatekno reports that one file is deleted:

+-------------------------------------------------------------------+
| Repo                            branch     ±origin  U  D  M  C  A |
+===================================================================+
[...]
| template-keynote-UiT            main             0  0  1  0  0  0 |
[...]
| ================                                                  |
| Repo Count                      Dirty   fetch/push  U  D  M  C  A |
| 33                              2                0  1  1  0  0  0 |
+-------------------------------------------------------------------+

But when checking with git, nothing seems to be missing or deleted:

cd divvungiellatekno/template-keynote-UiT/
template-keynote-UiT on  main 
➜ git status
On branch main
Your branch is up to date with 'origin/main'.

nothing to commit, working tree clean

So what's going on?

[snomos]

Investigation: Why gut status and git status show different results for template-keynote-UiT

This analysis was done by Claude.

Problem

gut status -o divvungiellatekno shows that the repository template-keynote-UiT contains a deleted file (D=1), but git status in the same repository shows everything is clean.

Investigation

1. gut status shows:

   | template-keynote-UiT            main             0  0  1  0  0  0 |
   

   (D column = 1, indicating a deleted file)

2. git status shows:

   On branch main
   Your branch is up to date with 'origin/main'.
   nothing to commit, working tree clean
   

3. Git index status:

   $ git ls-files --stage
   100755 e4e141d4cfb60eb9f01b3308e90dfd92697356df 0       UiT-templat-nynorsk-blått.kth
   100755 e4e141d4cfb60eb9f01b3308e90dfd92697356df 0       UiT-templat-nynorsk-blått.kth

   The file is registered twice in the git index with the same hash!

4. Actual file status:

   $ ls -lh "UiT-templat-nynorsk-blått.kth"
   -rwx------@ 1 smo036  staff   1.7M Jan 30  2024 UiT-templat-nynorsk-blått.kth

   The file exists on disk.

5. git2 library (used by gut):

   • Sees 1 status entry: Status(WT_DELETED) for the file
   • Interprets the duplicate index entry as a deleted file

Root Cause

The problem is a corrupted git index with a duplicate entry for the same file. This happened at some point in the past, likely during an unusual git operation or conflict resolution.

Why git and gut behave differently:

• Git (command line, version 2.52.0): Has logic that handles or ignores duplicate index entries and reports the repository as clean
• libgit2/git2-rs (used by gut): Interprets duplicate index entries as a sign that the file is deleted in the working directory, and reports WT_DELETED status

Solution

Run this command in the repository to fix the corrupted index:

cd ~/langtech/gut/divvungiellatekno/template-keynote-UiT
git rm --cached "UiT-templat-nynorsk-blått.kth"
git add "UiT-templat-nynorsk-blått.kth"

This will remove the duplicate entry and add the file again with only one entry in the index.

Conclusion

This is not a bug in gut, but rather a corrupted git index in the repository that is interpreted differently by git and libgit2. gut is actually behaving more correctly by reporting that something is wrong, while the git command line masks the problem.

[snomos]

Root Cause: Unicode Normalization Conflict (NFD vs NFC)

This analysis was done by Claude.

The Real Problem

The issue was not a corrupted git index, but rather a Unicode normalization conflict between macOS and Git history.

The filename UiT-templat-nynorsk-blått.kth contains the letter å, which can be represented in two different Unicode forms:

• NFD (Normalized Form Decomposed): å = a (U+0061) + combining ring above (U+030A)
• NFC (Normalized Form Composed): å = single character (U+00E5)

What Was Happening

1. macOS filesystem (APFS/HFS+) automatically converts all filenames to NFD normalization
2. Git history contained both variants of the same filename:
   • NFD version: bla\u030att (decomposed å)
   • NFC version: bl\u00E5tt (composed å)
3. Git treated these as two different files with identical visual appearance
4. macOS filesystem only had one physical file (the NFD version)
5. libgit2/git2 (used by gut) saw that the NFC version was missing from disk and reported it as deleted (WT_DELETED)
6. Command-line git has special logic to handle this situation and ignores the discrepancy

Verification

Hex dump of git index confirmed the duplicate:

626c 61cc 8a74 74    # NFD: bla + combining-ring + tt
626c c3a5 7474       # NFC: bl + precomposed-å + tt

Solution Applied

Created a new commit that removes the NFC duplicate, keeping only the NFD version (which matches macOS filesystem behavior):

# Created a corrected tree with only NFD version
printf "100755 blob e4e141d4cfb60eb9f01b3308e90dfd92697356df\tUiT-templat-nynorsk-bla\xcc\x8att.kth\n" | git mktree
# Created new commit with fixed tree
git commit-tree <tree-hash> -p HEAD -m "Fix: Remove NFC duplicate of filename with NFD normalization"
# Moved HEAD to new commit
git reset --hard <commit-hash>
# Force-pushed to remote
git push --force-with-lease

Result

After the fix:

• ✅ Only 1 entry in git index (NFD normalized)
• ✅ File exists on filesystem (NFD normalized by macOS)
• ✅ git status shows clean working tree
• ✅ gut status no longer shows deleted file (D=0)

Conclusion

This was not a bug in gut - gut was correctly reporting the state of the repository according to libgit2. The issue was a Unicode normalization conflict in the git history, likely introduced when the file was added from a non-macOS system (Linux/Windows) that uses NFC normalization, while macOS uses NFD.

[snomos]

Proposals to Prevent Unicode Normalization Issues in the Future

Suggestions by Claude.

Now that we understand the root cause (NFD vs NFC normalization conflicts), here are six proposals to prevent similar issues going forward:

1. Git Configuration: core.precomposeUnicode

Git has a built-in setting to handle this on macOS:

git config --global core.precomposeUnicode true

What this does:

• Converts filenames from NFD (macOS default) to NFC before storing in git
• Makes git behave more consistently across platforms
• This is actually default in newer Git versions on macOS, but may be disabled in older installations

Limitation:

• Must be set on all macOS systems working with the repo
• Existing history is not changed, only new commits

2. Pre-commit Hook to Validate Filenames

Create a pre-commit hook that checks for NFD/NFC inconsistencies:

#!/bin/bash
# .git/hooks/pre-commit

# Check for files with non-NFC normalized names
git diff --cached --name-only -z | while IFS= read -r -d '' file; do
    # Convert to NFC and check if it differs
    nfc_name=$(echo "$file" | iconv -f UTF-8-MAC -t UTF-8)
    if [ "$file" != "$nfc_name" ]; then
        echo "Error: File '$file' has NFD-normalized name"
        echo "Please rename to NFC form: '$nfc_name'"
        exit 1
    fi
done

3. CI/CD Validation

Add a test in CI/CD pipelines that checks for duplicates:

# Check for duplicate entries in git index
duplicates=$(git ls-files --stage | awk '{print $4}' | sort | uniq -d)
if [ -n "$duplicates" ]; then
    echo "Error: Duplicate entries found in git index:"
    echo "$duplicates"
    exit 1
fi

4. Improve gut to Warn About the Issue

Add a warning in gut when it detects NFD/NFC conflicts instead of just reporting "deleted":

// In git/status.rs
if Status::is_wt_deleted(status) {
    if let Some(path) = entry.path() {
        // Check if there's a potential NFD/NFC conflict
        let nfc_path = path.to_string();
        // Check if filesystem has NFD variant
        
        if filesystem_has_nfd_variant(&nfd_path) {
            eprintln!("Warning: '{}' may have Unicode normalization conflict (NFD/NFC)", path);
            eprintln!("  This is common on macOS. Run: git config core.precomposeUnicode true");
        }
        deleted.push(path.to_string());
    }
}

5. Documentation

Create a FAQ or troubleshooting guide for gut that explains:

• Unicode normalization issues on macOS
• How to set core.precomposeUnicode
• How to fix existing conflicts

6. Automatic Check in gut clone

Add a check in gut clone that automatically sets core.precomposeUnicode on macOS:

#[cfg(target_os = "macos")]
fn configure_macos_git(repo_path: &Path) {
    Command::new("git")
        .args(&["config", "core.precomposeUnicode", "true"])
        .current_dir(repo_path)
        .output()
        .ok();
}

---

Recommended Combination

Best approach:

1. ✅ Set git config --global core.precomposeUnicode true on all macOS machines
2. ✅ Add a gut doctor command that checks this setting and warns about inconsistencies
3. ✅ Document the issue in README/Wiki
4. ✅ Consider adding automatic configuration in gut init/gut clone on macOS

Thoughts? Which proposals should we prioritize?

[snomos]

@dylanhand wdyt? @albbas? @Phaqui? @flammie?

[snomos]

Update: Corrected Fix Applied

Update by Claude.

Issue with Previous Fix

My previous fix was incorrect. I created a commit with NFD normalization, but when core.precomposeUnicode=true is set (as recommended), Git expects tree objects to contain NFC-normalized filenames.

What went wrong:

1. First fix: Created tree with NFD: bla\u030att (decomposed)
2. User set core.precomposeUnicode=true (recommended setting)
3. Git automatically converts NFD (from macOS filesystem) to NFC when reading
4. Git looked for bl\u00E5tt (NFC) in tree object
5. Tree contained bla\u030att (NFD) - mismatch!
6. Git reported file as "untracked"

Corrected Fix

Created a new commit with NFC normalization instead:

# Create tree with NFC-normalized filename
printf "100755 blob e4e141d4cfb60eb9f01b3308e90dfd92697356df\tUiT-templat-nynorsk-bl\xc3\xa5tt.kth\n" | git mktree
# Create new commit based on HEAD^ (before wrong NFD commit)
git commit-tree <tree-hash> -p HEAD^ -m "Fix: Use NFC normalization for filename (compatible with core.precomposeUnicode)"
git reset --hard <commit-hash>
git push --force-with-lease

Why NFC is Correct

When core.precomposeUnicode=true:

• Git automatically converts NFD (from macOS filesystem) to NFC when reading files
• Git expects tree objects to contain NFC-normalized filenames
• This makes Git work consistently across macOS, Linux, and Windows

Current State

After the corrected fix:

• ✅ Git tree contains NFC: bl\u00E5tt (composed)
• ✅ macOS filesystem has NFD: bla\u030att (decomposed)
• ✅ core.precomposeUnicode=true converts between them automatically
• ✅ git status shows clean working tree
• ✅ gut status shows no issues (0 deleted, 0 untracked, 0 modified)

Lesson Learned

The correct approach is:

• Always use NFC normalization in Git commits (tree objects, index)
• Let core.precomposeUnicode=true handle the conversion between NFC (Git) and NFD (macOS filesystem)
• This is why core.precomposeUnicode=true is the default on modern Git for macOS

[snomos]

Scope of the Problem: Widespread NFD Normalization Issues

Analysis by Claude.

Discovery

After investigating other repositories in the organization, we've discovered that the Unicode normalization issue is much more widespread than initially thought.

Scale:

• Multiple repositories are affected by NFD/NFC normalization conflicts
• Some repositories contain over 1,000 files with combining characters in their filenames
• This affects approximately 60%+ of files in some repositories
• The issue particularly affects repositories with filenames containing:
  • Non-ASCII characters with diacritics
  • Cyrillic characters with combining marks
  • Other Unicode characters that can be represented in both NFD and NFC forms

Impact

With core.precomposeUnicode=true (recommended setting):

• git CLI shows these files as "modified" (due to special handling logic)
• libgit2/gut reports them as "deleted" (stricter interpretation)
• This creates confusion and makes gut status less useful for affected repositories

Proposed Solutions

We need tooling to detect and fix these issues. Here are the options:

Option 1: Integrate into gut (Recommended for Detection)

A. gut doctor command (General health check)

gut doctor -o <org>
# Checks:
# - core.precomposeUnicode setting
# - NFD/NFC conflicts in all repos
# - Other potential issues (extensible)

B. gut check-unicode command (Specialized)

gut check-unicode -o <org>
# Shows:
# - Repos with NFD filenames in tree objects
# - Number of affected files
# - Suggested fixes

C. Warning in gut status (Lightweight)

Warning: 14 files may have Unicode normalization conflicts
  Run: gut check-unicode for details

Advantages:

• Users already have gut installed
• Natural place for git-related checks
• Can integrate into existing workflows
• Can run checks automatically during certain operations

Option 2: Standalone Tool (Recommended for Fixing)

Name: git-unicode-doctor or git-nfc-fixer

# Check a repo
git-unicode-doctor check

# Fix a repo
git-unicode-doctor fix --dry-run
git-unicode-doctor fix --force

# Scan multiple repos
git-unicode-doctor scan ~/langtech/gut/*

Advantages:

• Can be used by anyone, not just gut users
• Easier to maintain separately
• Focused on one task
• Safer to have destructive operations (force-push) in separate tool

Recommended Approach

Combination of both:

1. gut doctor command - for detection and warnings

   • Easy to use for existing gut users
   • Can check many repos quickly
   • Can be integrated into CI/CD pipelines
   • Provides visibility into the problem

2. Standalone git-unicode-doctor - for fixing

   • Can be used on individual repos without gut
   • Safer to have a separate tool for destructive operations
   • Can be used by others outside Divvun/Giella environment
   • More careful and controlled fix process

Next Steps

Before implementing, we should decide:

1. Should we prioritize detection (gut doctor) or fixing (git-unicode-doctor) first?
2. Should fixing be automated or require manual review for each repository?
3. For repos with 1000+ affected files, should we batch-fix or provide migration guidance?

Thoughts on this approach?

[flammie]

I agree on using NFC.

If we can setup something server side that requires no user setup it can be useful, client side commit hooks and extra programs tend to have limited success and probably block innocent committers from working,

Gut doing sanity checks on config values and such sounds like a good idea, if it gets implemented it should just as well be ran unconditionally on every invocation.

Some doctoring commands sound useful too.

[snomos]

NFC means "maximally precomposed letters", which is not the same as precomposed letters only. As we know very well there are several languages in our infra that use letters that can only be represented as base+combining diacritic. The question is then how do we most efficiently ensure that NFC is what is stored internally in the git database, so that all OSs can work transparently with all filenames?

As git nowadays uses core.precomposeUnicode=true as default on macOS, we can assume it is set client-side. This is also all that should be required client-side. Cf the following:

The default for this config setting on OS X is true and there should be no reason to override it.

(from: https://www.git-tower.com/help/guides/faq-and-tips/faq/unicode-filenames/mac)

So, as Flammie and Claude suggest, we continue with:

• add sanity checks to gut for config values
• add error detection command to gut (ie filename check for NFC - it is probably going to run for a while on each invocation, so should not be automatically run on other command invocations)
• make an independent tool to fix filename encoding (NFC) errors in the git database

@dylanhand @albbas @Phaqui WDYT?

[dylanhand]

Agree it'd be a good idea to have detection built into gut.

Also reasonable to have a separate tool for fixing this. It seems unlikely that we're the first to encounter this issue so I wonder if something already exists. I had a quick search now but so far haven't found anything that looks excellent.

[dylanhand]

Ah here's a library written in rust for normalizing to NFC: https://github.com/cr0sh/jaso

[snomos]

It is for normalising filenames, not fix the issue in the git index/database. But it is good to know, and can surely be used.

[snomos]

Agree it'd be a good idea to have detection built into gut.

Done, see work in this issue (which has been turned into a PR).

Also reasonable to have a separate tool for fixing this. It seems unlikely that we're the first to encounter this issue so I wonder if something already exists. I had a quick search now but so far haven't found anything that looks excellent.

Also done and tested. Seems to work correctly on the cases I tested it on. See this repo: https://github.com/divvun/nfd-fixer

[snomos]

This PR (was: issue) is now ready for review. What I have done (with a lot of help from Claude) is:

• create a new gut health command that includes the following:
• ✅ detection of NFD/NFC normalisation issues in the git index (NFC is the only form that should be in the index)
  • created the tool nfd-fixer to actually fix these issues
• ✅ detection of case-duplicate filenames
• ✅ checking whether Git LFS is installed
• ✅ detection of large files not stored in Git LFS, but in regular Git; default threshold: 50 MB
• ✅ detection of long filenames/paths to avoid problems on some systems
  • filename length default threshold: 200 bytes
  • path length default threshold: 400 bytes
• ✅ checking Git version
• ✅ checking core.precomposeUnicode on macOS
• ✅ checking core.autocrlf on Unix-like systems

The various default thresholds can be changed with options to the new command.

When issues are detected, recommended actions are suggested in the report printed.

Comments and reviews welcome 🙂

@dylanhand @albbas @Phaqui @flammie

When this is done and merged, the next step is to re-add gut to CI, to ensure we have up-to-date binaries that people can download and install.