[PATCH v2 0/3] Additional FAQ entries

This series introduces some additional Git FAQ entries on various
topics.  They are all things I've seen in my professional life or on
Stack Overflow, so I've written documentation.

I've opted not to include backing up repositories in the syncing patch
because I think they're separate topics.  We could well end up with an
additional FAQ entry on that topic, which is left as an exercise for the
reader.

Changes from v1:
* Drop the monorepo patch for now; I want to revise it further.
* Reorder the working tree patch to place more warnings up front.
* Mention core.gitproxy and socat.
* Rephrase text in the EOL entry to read correctly and be easier to
  understand.
* Improve the commit message for the working tree FAQ entry to make it
  clearer that users wish to transfer uncommitted changes.

brian m. carlson (3):
  gitfaq: add documentation on proxies
  gitfaq: give advice on using eol attribute in gitattributes
  gitfaq: add entry about syncing working trees

 Documentation/gitfaq.txt | 90 +++++++++++++++++++++++++++++++++++++---
 1 file changed, 84 insertions(+), 6 deletions(-)

Many corporate environments and local systems have proxies in use.  Note
the situations in which proxies can be used and how to configure them.
At the same time, note what standards a proxy must follow to work with
Git.  Explicitly call out certain classes that are known to routinely
have problems reported various places online, including in the Git for
Windows issue tracker and on Stack Overflow, and recommend against the
use of such software.

Signed-off-by: brian m. carlson <redacted>
---
 Documentation/gitfaq.txt | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)

diff --git a/Documentation/gitfaq.txt b/Documentation/gitfaq.txt
index 946691c153..5c21951f7b 100644
--- a/Documentation/gitfaq.txt
+++ b/Documentation/gitfaq.txt

@@ -241,6 +241,29 @@ How do I know if I want to do a fetch or a pull?::
 	ignore the upstream changes.  A pull consists of a fetch followed
 	immediately by either a merge or rebase.  See linkgit:git-pull[1].
 
+[[proxy]]
+Can I use a proxy with Git?::
+	Yes, Git supports the use of proxies.  Git honors the standard `http_proxy`,
+	`https_proxy`, and `no_proxy` environment variables commonly used on Unix, and
+	it also can be configured with `http.proxy` and similar options for HTTPS (see
+	linkgit:git-config[1]).  The `http.proxy` and related options can be
+	customized on a per-URL pattern basis.  In addition, Git can in theory
+	function normally with transparent proxies that exist on the network.
++
+For SSH, Git can support a proxy using `core.gitproxy`. Commonly used tools
+include `netcat` and `socat`.  However, they must be configured not to exit when
+seeing EOF on standard input, which usually means that `netcat` will require
+`-q` and `socat` will require a timeout with something like `-t 10`.
++
+Note that in all cases, for Git to work properly, the proxy must be completely
+transparent.  The proxy cannot modify, tamper with, change, or buffer the
+connection in any way, or Git will almost certainly fail to work.  Note that
+many proxies, including many TLS middleboxes, Windows antivirus and firewall
+programs other than Windows Defender and Windows Firewall, and filtering proxies
+fail to meet this standard, and as a result end up breaking Git.  Because of the
+many reports of problems, we recommend against the use of these classes of
+software and devices.
+
 Design
 ------
 

Users very commonly want to sync their working tree with uncommitted
changes across machines, often to carry across in-progress work or
stashes.  Despite this not being a recommended approach, users want to
do it and are not dissuaded by suggestions not to, so let's recommend a
sensible technique.

The technique that many users are using is their preferred cloud syncing
service, which is a bad idea.  Users have reported problems where they
end up with duplicate files that won't go away (with names like "file.c
2"), broken references, oddly named references that have date stamps
appended to them, missing objects, and general corruption and data loss.
That's because almost all of these tools sync file by file, which is a
great technique if your project is a single word processing document or
spreadsheet, but is utterly abysmal for Git repositories because they
don't necessarily snapshot the entire repository correctly.  They also
tend to sync the files immediately instead of when the repository is
quiescent, so writing multiple files, as occurs during a commit or a gc,
can confuse the tools and lead to corruption.

We know that the old standby, rsync, is up to the task, provided that
the repository is quiescent, so let's suggest that and dissuade people
from using cloud syncing tools.  Let's tell people about common things
they should be aware of before doing this and that this is still
potentially risky.  Additionally, let's tell people that Git's security
model does not permit sharing working trees across users in case they
planned to do that.  While we'd still prefer users didn't try to do
this, hopefully this will lead them in a safer direction.

Signed-off-by: brian m. carlson <redacted>
---
 Documentation/gitfaq.txt | 47 ++++++++++++++++++++++++++++++++++++++--
 1 file changed, 45 insertions(+), 2 deletions(-)

diff --git a/Documentation/gitfaq.txt b/Documentation/gitfaq.txt
index ae1b526565..e5dab89d6c 100644
--- a/Documentation/gitfaq.txt
+++ b/Documentation/gitfaq.txt

@@ -83,8 +83,8 @@ Windows would be the configuration `"C:\Program Files\Vim\gvim.exe" --nofork`,
 which quotes the filename with spaces and specifies the `--nofork` option to
 avoid backgrounding the process.
 
-Credentials
------------
+Credentials and Transfers
+-------------------------
 
 [[http-credentials]]
 How do I specify my credentials when pushing over HTTP?::

@@ -185,6 +185,49 @@ Then, you can adjust your push URL to use `git@example_author` or
 `git@example_committer` instead of `git@example.org` (e.g., `git remote set-url
 git@example_author:org1/project1.git`).
 
+[[sync-working-tree]]
+How do I sync a working tree across systems?::
+	First, decide whether you want to do this at all.  Git usually works better
+	when you push or pull your work using the typical `git push` and `git fetch`
+	commands and isn't designed to share a working tree across systems.  This is
+	potentially risky and in some cases can cause repository corruption or data
+	loss.
++
+Usually, doing so will cause `git status` to need to re-read every file in the
+working tree.  Additionally, Git's security model does not permit sharing a
+working tree across untrusted users, so it is only safe to sync a working tree
+if it will only be used by a single user across all machines.
++
+It is important not to use a cloud syncing service to sync any portion of a Git
+repository, since this can cause corruption, such as missing objects, changed
+or added files, broken refs, and a wide variety of other corruption.  These
+services tend to sync file by file and don't understand the structure of a Git
+repository.  This is especially bad if they sync the repository in the middle of
+it being updated, since that is very likely to cause incomplete or partial
+updates and therefore data loss.
++
+Therefore, it's better to push your work to either the other system or a central
+server using the normal push and pull mechanism.  However, this doesn't always
+preserve important data, like stashes, so some people prefer to share a working
+tree across systems.
++
+If you do this, the recommended approach is to use `rsync -a --delete-after`
+(ideally with an encrypted connection such as with `ssh`) on the root of
+repository.  You should ensure several things when you do this:
++
+* There are no additional worktrees enabled for your repository.
+* You are not using a separate Git directory outside of your repository root.
+* You are comfortable with the destination directory being an exact copy of the
+	source directory, _deleting any data that is already there_.
+* The repository is in a quiescent state for the duration of the transfer (that
+	is, no operations of any sort are taking place on it, including background
+	operations like `git gc` and operations invoked by your editor).
++
+Be aware that even with these recommendations, syncing in this way has some risk
+since it bypasses Git's normal integrity checking for repositories, so having
+backups is advised.  You may also with to do a `git fsck` to verify the
+integrity of your data on the destination system after syncing.
+
 Common Issues
 -------------
 

In the FAQ, we tell people how to use the text attribute, but we fail to
explain what to do with the eol attribute.  As we ourselves have
noticed, most shell implementations do not care for carriage returns,
and as such, people will practically always want them to use LF endings.
Similar things can be said for batch files on Windows, except with CRLF
endings.

Since these are common things to have in a repository, let's help users
make a good decision by recommending that they use the gitattributes
file to correctly check out the endings.

In addition, let's correct the cross-reference to this question, which
originally referred to "the following entry", even though a new entry
has been inserted in between.  The cross-reference notation should
prevent this from occurring and provide a link in formats, such as HTML,
which support that.

Signed-off-by: brian m. carlson <redacted>
---
 Documentation/gitfaq.txt | 20 ++++++++++++++++----
 1 file changed, 16 insertions(+), 4 deletions(-)

diff --git a/Documentation/gitfaq.txt b/Documentation/gitfaq.txt
index 5c21951f7b..ae1b526565 100644
--- a/Documentation/gitfaq.txt
+++ b/Documentation/gitfaq.txt

@@ -406,8 +406,9 @@ I'm on Windows and git diff shows my files as having a `^M` at the end.::
 +
 You can store the files in the repository with Unix line endings and convert
 them automatically to your platform's line endings.  To do that, set the
-configuration option `core.eol` to `native` and see the following entry for
-information about how to configure files as text or binary.
+configuration option `core.eol` to `native` and see
+<<recommended-storage-settings,the question on recommended storage settings>>
+for information about how to configure files as text or binary.
 +
 You can also control this behavior with the `core.whitespace` setting if you
 don't wish to remove the carriage returns from your line endings.

@@ -469,14 +470,25 @@ references, URLs, and hashes stored in the repository.
 +
 We also recommend setting a linkgit:gitattributes[5] file to explicitly mark
 which files are text and which are binary.  If you want Git to guess, you can
-set the attribute `text=auto`.  For example, the following might be appropriate
-in some projects:
+set the attribute `text=auto`.
++
+With text files, Git will generally ensure that LF endings are used in the
+repository, and will honor `core.autocrlf` and `core.eol` to decide what options
+to use when checking files out.  You can also override this by specifying a
+particular line ending such as `eol=lf` or `eol=crlf` if those files must always
+have that ending in the working tree (e.g., for functionality reasons).
++
+For example, the following might be appropriate in some projects:
 +
 ----
 # By default, guess.
 *	text=auto
 # Mark all C files as text.
 *.c	text
+# Ensure all shell files end up with LF endings and all batch files end up
+# with CRLF endings in the working tree and both end up with LF in the repo.
+*.sh text eol=lf
+*.bat text eol=crlf
 # Mark all JPEG files as binary.
 *.jpg	binary
 ----

On Sun, Nov 7, 2021 at 5:55 PM brian m. carlson
[off-list ref] wrote:

Many corporate environments and local systems have proxies in use.  Note
the situations in which proxies can be used and how to configure them.
At the same time, note what standards a proxy must follow to work with
Git.  Explicitly call out certain classes that are known to routinely
have problems reported various places online, including in the Git for
Windows issue tracker and on Stack Overflow, and recommend against the
use of such software.

A couple minor comments; may not be worth a reroll or discussion...

I've seen this come up on the mailing list a couple times recently,
though I haven't really followed along and don't use Git through an
SSH proxy, thus I did have to go do some reading to understand what
this is talking about. Perhaps people searching out this FAQ entry
will already have sufficient context to understand what this is
saying, so maybe no additional context is needed here. However, I was
wondering if it might make sense for this to give a bit of reason
explaining _why_ these tools need to be configured to not exit
immediately upon EOF. As it stands now, this solution is a black box;
it will work but people won't understand why. Perhaps that doesn't
matter since most people consulting a FAQ like this probably just want
to get the thing working and don't care about the underlying details.
Then again, if the underlying reason is made more readily apparent,
maybe this knowledge can become more widespread.

+Note that in all cases, for Git to work properly, the proxy must be completely
+transparent.  The proxy cannot modify, tamper with, change, or buffer the
+connection in any way, or Git will almost certainly fail to work.  Note that

"modify, tamper with, change" sounds like it came from the Department
of Redundancy Department. I like the sound of "tamper with" since the
image it conveys feels quite suitable here. Perhaps this could be
simplified to:

   The proxy cannot tamper with or buffer the...

+many proxies, including many TLS middleboxes, Windows antivirus and firewall
+programs other than Windows Defender and Windows Firewall, and filtering proxies
+fail to meet this standard, and as a result end up breaking Git.  Because of the
+many reports of problems, we recommend against the use of these classes of
+software and devices.

On Sun, Nov 7, 2021 at 5:55 PM brian m. carlson
[off-list ref] wrote:

I like the concrete explanation in the commit message of why `.sh` and
`.bat` files need to be configured specially, and was expecting that
quite useful information to be repeated here in the body. That does
seem exactly like what someone would come to the FAQ searching for,
and even though the above content hints at it, the hint may be just
vague enough to be overlooked. Consequently, I'm wondering if it ought
to be spelled out more explicitly here. In particular, if we give
examples of actual error messages people might encounter when
attempting to run a CRLF shell script or an LF batch file, that would
really give a FAQ searcher/reader something to latch onto when trying
to solve a problem. This could be done as a lead-in paragraph
immediately before the "For example, the following...".

Also, following v1 review, I think you had intended[1]: s/end up with/have/

[1]: https://lore.kernel.org/git/YW9wgbN%2Fb8NkVp4z@camp.crustytoothpaste.net/

On Sun, Nov 7, 2021 at 5:55 PM brian m. carlson
[off-list ref] wrote:

The phrase "usually works better" makes this statement feel weak, thus
it may not convey the potential severity of the issue. I wonder if
rewording it something like this would make the statement more
forceful:

    Git works best when you `git push` and `git pull` your work
    between machines; it is not designed to share a working tree
    across systems. [...]

+Usually, doing so will cause `git status` to need to re-read every file in the
+working tree.  Additionally, Git's security model does not permit sharing a
+working tree across untrusted users, so it is only safe to sync a working tree
+if it will only be used by a single user across all machines.
++
+It is important not to use a cloud syncing service to sync any portion of a Git
+repository, since this can cause corruption, such as missing objects, changed
+or added files, broken refs, and a wide variety of other corruption.  These
+services tend to sync file by file and don't understand the structure of a Git
+repository.  This is especially bad if they sync the repository in the middle of
+it being updated, since that is very likely to cause incomplete or partial
+updates and therefore data loss.

Taking into consideration that people who are experiencing such
corruption will likely include the name of the syncing service in
their search query, would it make sense to mention some well-known
services here in order to make it more likely that people will
actually find this entry? Something like this, perhaps:

    It is important not to use a cloud syncing service (such as DropBox,
    FooBar, CowMoo, BuzzingBee, etc.) to sync any portion of a Git
    repository...

+Therefore, it's better to push your work to either the other system or a central
+server using the normal push and pull mechanism.  However, this doesn't always
+preserve important data, like stashes, so some people prefer to share a working
+tree across systems.
++
+If you do this, the recommended approach is to use `rsync -a --delete-after`
+(ideally with an encrypted connection such as with `ssh`) on the root of
+repository.  You should ensure several things when you do this:
++
+* There are no additional worktrees enabled for your repository.

I don't fully understand this restriction. Can you explain it (at
least here in the email discussion)?

+* You are not using a separate Git directory outside of your repository root.

Same question about this restriction.

+* You are comfortable with the destination directory being an exact copy of the
+       source directory, _deleting any data that is already there_.
+* The repository is in a quiescent state for the duration of the transfer (that
+       is, no operations of any sort are taking place on it, including background
+       operations like `git gc` and operations invoked by your editor).
++
+Be aware that even with these recommendations, syncing in this way has some risk
+since it bypasses Git's normal integrity checking for repositories, so having
+backups is advised.  You may also with to do a `git fsck` to verify the
+integrity of your data on the destination system after syncing.

s/with/wish/

In fact, as with "usually" above, "wish" may be too weak. Perhaps say
instead that it is "recommended" that you use `git fsck` to verify the
integrity.

On Sun, Nov 7, 2021 at 5:55 PM brian m. carlson
[off-list ref] wrote:

This series introduces some additional Git FAQ entries on various
topics.  They are all things I've seen in my professional life or on
Stack Overflow, so I've written documentation.

Changes from v1:
* Drop the monorepo patch for now; I want to revise it further.
* Reorder the working tree patch to place more warnings up front.
* Mention core.gitproxy and socat.
* Rephrase text in the EOL entry to read correctly and be easier to
  understand.
* Improve the commit message for the working tree FAQ entry to make it
  clearer that users wish to transfer uncommitted changes.

Thanks for re-rolling. FAQ entries are, of course, bikeshedding
fodder, so I left a number of bikeshedding comments and pointed out
one minor typo. Whether any of the comments warrant a re-roll or are
even worth discussing is a matter of opinion, and they could certainly
all be addressed later (by someone) atop the current series if
desired; I don't think any of the comments were blockers.

On 2021-11-07 at 23:27:24, Eric Sunshine wrote:

I've seen this come up on the mailing list a couple times recently,
though I haven't really followed along and don't use Git through an
SSH proxy, thus I did have to go do some reading to understand what
this is talking about. Perhaps people searching out this FAQ entry
will already have sufficient context to understand what this is
saying, so maybe no additional context is needed here. However, I was
wondering if it might make sense for this to give a bit of reason
explaining _why_ these tools need to be configured to not exit
immediately upon EOF. As it stands now, this solution is a black box;
it will work but people won't understand why. Perhaps that doesn't
matter since most people consulting a FAQ like this probably just want
to get the thing working and don't care about the underlying details.
Then again, if the underlying reason is made more readily apparent,
maybe this knowledge can become more widespread.

I'll try to see if I can stuff in a sentence there about why that's
necessary.  I think I understand it sufficiently well to summarize it.

"modify, tamper with, change" sounds like it came from the Department
of Redundancy Department. I like the sound of "tamper with" since the
image it conveys feels quite suitable here. Perhaps this could be
simplified to:

   The proxy cannot tamper with or buffer the...

I realize this sounds redundant, but I'm trying to avoid the situation
where people say, "I'm not _tampering_ with it, since I'm authorized to
do this by the company.  I'm just modifying it to remove this
inappropriate content/malware/data leak."  My goal here is to make it
crystal clear that if you do this, you'll break things, and provide
ammunition for people to go to their IT departments and say, "Look, your
proxy prevents me from doing my job.  The Git developers say so.  Fix
it."

I can drop one of "change" and "modify", though, since I think they're
synonyms.
-- 
brian m. carlson (he/him or they/them)
Toronto, Ontario, CA

On 2021-11-07 at 23:48:00, Eric Sunshine wrote:

I like the concrete explanation in the commit message of why `.sh` and
`.bat` files need to be configured specially, and was expecting that
quite useful information to be repeated here in the body. That does
seem exactly like what someone would come to the FAQ searching for,
and even though the above content hints at it, the hint may be just
vague enough to be overlooked. Consequently, I'm wondering if it ought
to be spelled out more explicitly here. In particular, if we give
examples of actual error messages people might encounter when
attempting to run a CRLF shell script or an LF batch file, that would
really give a FAQ searcher/reader something to latch onto when trying
to solve a problem. This could be done as a lead-in paragraph
immediately before the "For example, the following...".

I think that we could definitely add some text here to explain why one
would want to use this, providing an example.

Also, following v1 review, I think you had intended[1]: s/end up with/have/

I will fix that in v3.  Thanks for the reminder.
-- 
brian m. carlson (he/him or they/them)
Toronto, Ontario, CA

Hi brian,

On Sun, 7 Nov 2021, brian m. carlson wrote:

Many corporate environments and local systems have proxies in use.  Note
the situations in which proxies can be used and how to configure them.
At the same time, note what standards a proxy must follow to work with
Git.  Explicitly call out certain classes that are known to routinely
have problems reported various places online, including in the Git for

Not a big issue, but I think there is an "at" or "to" missing before
"various places online".

The first idea I had after reading this is: where are these examples
documented? Certainly not where I expected them, namely at
https://git-scm.com/docs/git-config#Documentation/git-config.txt-coregitProxy

And then I got puzzled. Why would `gitproxy` be used for the _SSH_
protocol? And I don't think it is used. I think it is only used for
connections using the `git://` protocol. I might very easily be wrong, of
course.

However, they must be configured not to exit when +seeing EOF on
standard input, which usually means that `netcat` will require +`-q` and
`socat` will require a timeout with something like `-t 10`.

IMHO it would make sense to add a concrete example, or maybe even two
concrete examples, one for `netcat` and one for `socat`.

++
+Note that in all cases, for Git to work properly, the proxy must be completely
+transparent.  The proxy cannot modify, tamper with, change, or buffer the
+connection in any way, or Git will almost certainly fail to work.  Note that
+many proxies, including many TLS middleboxes, Windows antivirus and firewall
+programs other than Windows Defender and Windows Firewall, and filtering proxies
+fail to meet this standard, and as a result end up breaking Git.  Because of the
+many reports of problems, we recommend against the use of these classes of
+software and devices.
+

This is good advice.

Ciao,
Dscho

Hi Eric,

On Sun, 7 Nov 2021, Eric Sunshine wrote:

On Sun, Nov 7, 2021 at 5:55 PM brian m. carlson
[off-list ref] wrote:

quoted

This series introduces some additional Git FAQ entries on various
topics.  They are all things I've seen in my professional life or on
Stack Overflow, so I've written documentation.

Changes from v1:
* Drop the monorepo patch for now; I want to revise it further.
* Reorder the working tree patch to place more warnings up front.
* Mention core.gitproxy and socat.
* Rephrase text in the EOL entry to read correctly and be easier to
  understand.
* Improve the commit message for the working tree FAQ entry to make it
  clearer that users wish to transfer uncommitted changes.

Thanks for re-rolling. FAQ entries are, of course, bikeshedding
fodder,

That is one possible perspective. Another possible perspective is that FAQ
entries are the first thing users in distress might turn to, so the more
helpful they are, the better. Maybe it would make sense to focus on that
a bit more. As in your suggestion to name cloud sync providers by name, to
make the FAQ entry eminently more "findable". To me, that did not at all
feel like pointless bike-shedding, but more like a really valuable
improvement that users will appreciate (even if they will be oblivious of
the fact that they appreciate it, of course).

Ciao,
Dscho

Hi brian,

On Sun, 7 Nov 2021, brian m. carlson wrote:

This series introduces some additional Git FAQ entries on various
topics.  They are all things I've seen in my professional life or on
Stack Overflow, so I've written documentation.

I've opted not to include backing up repositories in the syncing patch
because I think they're separate topics.  We could well end up with an
additional FAQ entry on that topic, which is left as an exercise for the
reader.

Makes sense.

You hinted in the thread that you were planning on submitting a v3. I hope
that my feedback regarding `core.gitproxy` is still in time for that.

Thank you,
Dscho

On 2022-01-04 at 13:54:52, Johannes Schindelin wrote:

Hi brian,

On Sun, 7 Nov 2021, brian m. carlson wrote:

quoted

This series introduces some additional Git FAQ entries on various
topics.  They are all things I've seen in my professional life or on
Stack Overflow, so I've written documentation.

I've opted not to include backing up repositories in the syncing patch
because I think they're separate topics.  We could well end up with an
additional FAQ entry on that topic, which is left as an exercise for the
reader.

Makes sense.

You hinted in the thread that you were planning on submitting a v3. I hope
that my feedback regarding `core.gitproxy` is still in time for that.

I can definitely include that in v3.  I'm hoping I can get to it later
this week or during the weekend before my schedule becomes busy again.

I appreciate the thoughtful comments.
-- 
brian m. carlson (he/him or they/them)
Toronto, Ontario, CA