Re: AI Textconv filter misconfiguration on Windows leads to silent corruption of diff output (ongoing investigation)

In short: PNG files were corrupted, but DLLs files not !

Hi Git developers,

I am submitting a detailed report about a serious issue I encountered on Windows where a globally configured Git textconv filter silently corrupts binary files during checkout. The corruption occurred without any warnings, and only became visible when comparing Git‑checked‑out files with the original versions downloaded via curl.

This report documents the cause, the technical mechanism, the reproduction steps, the fix, and the future risks.

---

Summary of the issue
A PNG file inside the repository https://github.com/openai/openai-openapi became corrupted after cloning. The same file downloaded via curl was intact. This immediately suggested that a Git filter was rewriting the file during checkout.

The root cause was a global .gitattributes file containing:

    * text diff=lfclean

Combined with the following global diff driver configuration:

    diff.lfclean.textconv = sed -e s/\r//
    diff.lfclean.binary = false

This configuration forces Git to treat *all* files as text, including binary formats, and to run a textconv filter that removes carriage return characters. Any binary file containing 0x0D bytes is silently modified during checkout.

---

Why PNG files were corrupted but DLL files were not
PNG files contain structured binary chunks (tEXt, iTXt, zTXt) that may legitimately include CR/LF characters. When the textconv filter removes CR bytes, the chunk lengths no longer match the actual data, resulting in a corrupted PNG.

DLL files, on the other hand, typically contain no CR characters at all. Because the filter only removes CR bytes, DLL files remain unchanged simply because there is nothing for the filter to remove. This makes the corruption appear “selective”, but it is purely accidental.

Any binary format containing CR bytes is at risk.

---

Reproduction steps
1. Create a global .gitattributes file:

       * text diff=lfclean

2. Configure the diff driver globally:

       diff.lfclean.textconv=sed -e s/\r//
       diff.lfclean.binary=false

3. Clone any repository containing binary files.

4. Compare the blob with the working-tree file:

       git show HEAD:path/to/file > raw.bin
       fc /b raw.bin path/to/file

If the binary contains CR bytes, the working-tree file will differ from the blob.

---

Cause of the problem
The pattern "*" matches every file.  
The attribute "text" forces Git to treat every file as text, overriding binary detection.  
The diff driver "lfclean" applies a textconv filter that rewrites file contents.  
Git applies this filter during checkout, not only during diff operations.

This combination guarantees corruption of any binary file containing CR bytes.

---

Solution
The correct fix is to remove the global .gitattributes file and the global diff driver:

    del C:\Users\<user>\.gitattributes
    git config --global --unset core.attributesfile
    git config --global --unset diff.lfclean.textconv
    git config --global --unset diff.lfclean.binary
    git config --global --remove-section diff.lfclean

Alternatively, restrict the filter to known text file extensions inside individual repositories.

---

Future risks
Global .gitattributes rules are applied to every repository on the system.  
Using "*" with "text" or any filter is extremely dangerous because:

- Git silently rewrites binary files during checkout.
- Corruption is not detected by Git.
- Corruption depends on file contents, making it unpredictable.
- Users may not realize that Git is modifying files.
- Any future repository containing binary formats with CR bytes will be corrupted.

This configuration effectively creates a system-wide corruption vector.

A warning or safeguard when "text" is forced globally might help prevent accidental misuse.

---

Conclusion
This issue demonstrates that global textconv filters can silently corrupt binary files on Windows. The corruption is subtle, unpredictable, and difficult to diagnose. Removing global filters and avoiding "*" patterns in global .gitattributes files is essential for data integrity.

Thank you for your time and consideration.

Best regards,
Skybuck