AI Textconv filter misconfiguration on Windows leads to silent corruption of diff output (ongoing investigation)

Dear Git maintainers,

I am writing to report a highly confusing and time‑consuming issue that I have encountered while using Git on Windows. The problem involves Git's textconv mechanism, the bundled sed, and a seemingly harmless configuration intended to remove carriage returns (CR) before displaying diffs. The issue is still under investigation, and I have not yet applied a definitive solution, but I believe it is worth reporting because it can cause massive confusion and wasted time for other users.

Background

I am working on a private branch of a Go project on Windows 10 (Git version 2.x, installed at C:\Tools\Git). I noticed that git diff between two commits (e.g., 429c244..70f57a8) showed added lines containing corrupt identifiers. For example:

- compareCache appeared as compaeCache
- return appeared as eturn
- from appeared as fom
- var appeared as va
- for appeared as fo
- cacheReader appeared as cacheReade
- CompareAndSwap appeared as CompaeAndSwap

The repository itself was clean. Extracting the actual file content from the commit with git show <commit>:net/sync_cache_reader.go correctly showed the proper spelling (e.g., compareCache). Running git diff --no-textconv produced the correct diff, proving that the corruption was introduced by a textconv filter.

Configuration

I had configured a textconv filter to normalize line endings before displaying diffs. Importantly, this configuration was not manually created by me; it was suggested by an AI assistant (specifically GitHub Copilot) while I was trying to solve a different problem with line endings. The AI recommended adding:

Global .gitconfig:
diff.lfclean.textconv=sed -e s/\r//

.gitattributes (in the repository):
*.go diff=lfclean

The intention was to remove carriage return (CR) characters from files before diffing, to avoid seeing ^M in the output.

This is a beautiful example of how AI can create confusion – the advice seemed perfectly reasonable but led to silent corruption of diffs, wasting many hours of debugging.

Observed Behavior

- git diff (with the filter active) shows corrupted output (missing the letter 'r').
- git diff --no-textconv shows correct output.
- git show <commit>:<file> shows correct content.
- git status shows no modifications; the working tree is clean.

Thus, the repository is not corrupt; the diff presentation is being altered.

Initial Diagnosis

I suspected that sed was misinterpreting the \r escape sequence. I found that Git for Windows bundles its own sed (at C:\Tools\Git\usr\bin\sed.exe), which is used even when sed is not in the system %PATH%. Running the command directly:

echo compareCache | C:\Tools\Git\usr\bin\sed.exe -e s/\r//

outputs:

compaeCache

So the command does strip the literal character 'r' instead of carriage returns. The likely reason is that the backslash before r is not preserved through the shell argument parsing on Windows; effectively, the expression becomes s/r//, which deletes all 'r' characters.

Impact

- Diff output becomes unreliable; users may falsely suspect repository corruption.
- Debugging is extremely time‑consuming. In my case, several hours were wasted, involving multiple tools and even AI assistants, before the root cause was identified.
- The problem is silent – no error messages are shown, making it hard to detect.
- This case also highlights a risk of relying on AI‑generated Git configurations without fully understanding the platform‑specific pitfalls.

Current Status

I have not yet decided on a permanent fix. I am considering removing the filter entirely, replacing it with a safer command (e.g., tr -d \r), or using --no-textconv when needed. However, I wanted to report this to the mailing list to:

1. Warn other Windows users about this pitfall, especially when taking advice from AI assistants.
2. Suggest possible improvements to Git to prevent such confusion in the future.

Suggested Improvements

- Documentation: Add a warning to gitattributes and git-config about using backslash escapes in textconv commands on Windows. Provide safe examples for removing CR, such as:
  diff.lfclean.textconv=tr -d \r
  or
  diff.lfclean.textconv=dos2unix

- Built-in filter: Consider offering a built-in textconv filter for line-ending normalization, e.g., diff.lfclean.textconv=git-crlf-remove, which would robustly handle CR stripping without relying on external tools or escaping pitfalls.

- Debugging aid: Add a flag like --debug-textconv that logs the exact command being executed for a textconv filter. This would help users see that their configured command may not be what they expect.

- Warning for suspicious patterns: On Windows, Git could detect textconv commands containing \r and emit a warning that this may be misinterpreted, suggesting safer alternatives.

Workaround for Affected Users

Remove the faulty filter:
git config --global --unset diff.lfclean.textconv
and delete or comment out the line in .gitattributes.

Alternatively, use git diff --no-textconv to bypass the filter when needed.

Conclusion

This issue is a result of a common misconfiguration combined with the quirks of Windows command parsing and the bundled sed. While Git itself is not at fault, better documentation and maybe a built-in solution would greatly improve the user experience for Windows developers. Additionally, this incident serves as a cautionary tale about relying on AI‑generated advice for system‑level configurations without understanding the underlying platform specifics.

I am happy to assist with testing any proposed documentation changes or additional debugging features. Thank you for your consideration.

Yours sincerely,
  Skybuck Flying (skybuck2000@hotmail.com)

Personal note: I BLAME LINUX FOR NOT FOLLOWING THE CARRIAGE RETURN NEW LINE CONVENTION. I ALSO BLAME/DISLIKE WINDOWS 11 ENVIRONMENT DIALOG PATH 2047 LIMIT WHICH MIGHT FURTHER CONFUSE THINGS, RE-ORDERING OF PATHS ALSO OCCURED BY AI TO TRY AND SOLVE THIS PATH DIALOG GUI LIMITATION ISSUE, LONGER PATH WAS SET DIRECTLY INTO THE REGISTRY.