Two crazy proposals for changing git's diff commands

So, here I am as a newly converted index-embracer---no more index
denying from me.

However, I'm still trying to wrap my brain around the various diff
commands that git provides and how they would fit into my workflow,
Junio and I have touched on this already in a previous thread, but I'm
starting here with more fresh and complete analysis of the UI around
diff.

The motivation for the *long* (my apologies) message below is largely
the fact that I realized my workflow would use the following command
very regularly:

	git diff-index --cached HEAD

I would be using this before every "got commit" to get a preview, and
it seems like a painfully long name for such a common operation. Even
its shortcut form:

	git diff --cached HEAD

is among the longest of the "git diff" shortcuts.

My other motivation is that in spite of having what I think is a
fairly good grasp of the structure of git, (the object database, the
index, etc.), I still have one heck of a time trying to remember which
diff commands are which.

So here we go...

Background
==========
At a conceptual level, there are 4 diff operations each of which acts
on an ordered pair (from->to) of trees. One operation takes two
explicit tree objects, while the other three act on 0 or 1 explicit
trees and 2 or 1 implicit trees (based on either the index or files in
the working directory). Specifically, these are tree->index,
index->files, and the composite of the two, tree->files.

To get at these 4 different operations, git provides 4 commands named
diff-tree, diff-index --cached, diff-files, and diff-index. On top of
these, git also provides some syntactic sugar in the form of "diff"
shortcuts for a total of 8 different diff commands.

This is all summarized in the following table:

Operation (from -> to)	Core command			Shortcut command
-----------------------	------------			---------------
diff <treeA> -> <treeB>	diff-tree <treeA> <treeB>	diff <treeA> <treeB>
diff <tree>  -> index	diff-index --cached <tree>	diff --cached <tree>
diff index   -> files	diff-files			diff
diff <tree>  -> files	diff-index <tree>		diff <tree>

I think this background is fairly complete, at least as far as the
functionality exposed by git-diff goes---I am ignoring git-diff-stages
for now, and throughout the remainder of this message.

Use cases
=========
After understanding things that far, I asked myself what each of the
four operations are useful for. The tree->tree case is easiest as it
simply shows the difference between two trees that exist in the object
store.

The remaining three cases are more interesting because they provide
mechanisms for querying trees that don't yet (or may never) exist in
the object store. Here are the questions I have been able to come up
with so far that the operations can help in answering:

<treeA> -> <treeB> (diff-tree <treeA> <treeB>, or diff <treeA> <treeB>)
  What changed between two trees?

<tree> -> index (diff-index --cached <tree>, diff --cached <tree>)
  When <tree> is HEAD: What will "git commit" do?

index -> files (diff-files, diff)
  What work have I done that I haven't updated into the index yet?
  Or, if not manually updating the index: What will "git commit -a" do?

<tree> -> files (diff-index <tree>, diff <tree>)
  When <tree> is HEAD: What will "git commit -a" do?

Can anyone else think of common use cases for these various
operations that I've missed?

Subjective comments/proposals
=============================
Everything above should be pretty much objective descriptions of how
things exist. From here on out, I'll start in with my opinions and
hopefully some useful proposals, (ordered by increasing likelihood of
being controversial).

It's interesting to me that both variants of diff-index (with and
without --cached) require a tree argument, while at the same time,
both variants seem most useful when used with HEAD. So it looks like
there's a reasonable default value for that option that is
missing. This should be pretty painless to fix, (no user retraining
required):

	Proposal 1: Make diff-index use HEAD if no <tree> is specified

	With this proposal, my most-common command now shrinks to:

		git diff-index --cached

	(And I was quite surprised to just learn that the shortcut
	version of "git diff --cached" already does default to HEAD
	rather than calling git-diff-files and erroring out on the
	unknown --cached argument. That's handy, even if a bit
	unexpected from the documentation of "git diff".)

So, my common command is a bit shorter, but I'd like to shrink it
more, and I still haven't addressed my which-diff-command-do-I-want
confusion.

First, I see a potential problem in the use cases table above. The
"git diff" command is taught in the tutorial as a way to preview what
will be committed by "commit -a". But I think this lays a trap for git
newcomers.

If "git diff" is learned as a commit preview, (during a larval
index-unaware stage), then this behavior will have to be unlearned
when the user starts using the index. At that point "git diff" becomes
a way to examine what will *not* be committed by "git commit" rather
than what *will* be committed by "git commit -a".

This seems an unkind thing to do to new users. Instead, users of
"commit -a" should be provided with a HEAD->files diff operation for
previewing commits. That's what they really want to see, (and not the
index->files diff that only happens to match in the case they haven't
manually dirtied the index).

So the 'correct' preview command for "commit -a" is currently one of
"diff HEAD" or "diff-index HEAD", and under Proposal 1 it would be
"diff-index".

So, without using the shortcut version, the tutorial's preview command
is down to a single name "diff-index", but it is rather awkward to
have a "-index" command in the index-avoidance stage of the tutorials.

And I think this goes toward my which-diff confusion. Consider the
three diff operations that allow for investigating un-committed trees:

	<tree>  -> index	diff-index --cached <tree>
	index   -> files	diff-files
	<tree>  -> files	diff-index <tree>

Here, "diff-index" is the only one of the three commands that does not
operate on the index. One can notice a similar thing in the ASCII
diagram from the core tutorial where diff-index is the only one of
these three operations that completely bypasses the index (!).

All that just to say that there's some inconsistent naming in the core
commands. This is papered over somewhat by the "git diff" shortcuts
but at the same time that also adds even _more_ diff commands for
people like me to have to learn.

So, here, finally is a proposal to change the names of some diff
commands. I expect this to be more controversial than proposal #1 as
it is sure to run up against ingrained muscle memory in some cases.
But I've tried to minimize that as much as possible, and I hope it
will be workable.

	Proposal #2: Provide the following 4 diff commands:

Operation (from -> to)	Proposal			   	
-----------------------	--------			   	
diff <treeA> -> <treeB>	diff <treeA> <treeA>		   	
diff <tree>  -> index	diff-index <tree> (default to HEAD)	
diff index   -> files	diff				   	
diff <tree>  -> files	diff-files <tree> (default to HEAD)	

The goal here is that using diff, diff-index, and diff-files without
any tree argument and without any options (such as --cached) should
cover the most common cases. So there's less typing in general.

These 3 diff commands can be presented as fundamental, and usable
without needing a layer of sugar above. So there is already less to
learn.n

Also, diff-index and diff-files have parallel structure and naming,
each performing a diff from HEAD or the given tree to either the
current index or files, respectively. So the names should also be
easier to learn.

Under this proposal, my "git commit" preview becomes:

	git diff-index

and the tutorial's "git commit -a" preview becomes:

	git diff-files

which looks pretty nice to me [*].

Let's examine the impact this proposal would have on the existing core
and shortcut diff commands. Here's an explanation of the Notes column
entries below:

"no change":	The existing command is provided in an identical form
		under the proposal.

"compatible":	The existing command will continue to function
		identically for backwards compatibility with muscle
		memory. But some things (such as --cached options)
		will simply be ignored and "unadvertised" under the
		new proposal

incompatible[*]:The proposal is not compatible with existing usage, so
		some amount of retraining will be needed. In each
		case, I've made notes on ways that might make this
		bearable.

Current core command	    After proposal	  Notes
--------------------	    --------------	  -----
diff-tree <treeA> <treeB>   diff <treeA> <treeB>  compatible
diff-index --cached <tree>  diff-index <tree>	  compatible
diff-files		    diff		  incompatible[1]
diff-index <tree>	    diff-files <tree>	  incompatible[2]

[1] Existing "diff-files" is index->files while the proposed
"diff-files" would be HEAD->files. Fortunately the retraining here is
to a simpler command ("diff") which already exists. Hopefully, current
users already prefer the simpler command anyway.

[2] Existing "diff-index <tree>" is <tree>->files while the proposed
"diff-index <tree>" would be <tree>->index. Fortunately the existing
"diff-index <tree>" has an existing shortcut as "diff <tree>" which
can be maintained compatibly. So hopefully, current users already
prefer the simpler command anyway. Otherwise, retraining for this
command will involves an index->files substitution, but hopefully the
consistent naming under the new proposal will help here.

Current shortcut command    After proposal	  Notes
------------------------    --------------	  -----
diff <treeA> <treeB>	    diff <treeA> <treeB>  no change
diff --cached <tree>	    diff-index <tree>	  compatible [3]		
diff			    diff		  no change
diff <tree>		    diff-files <tree>	  compatible [3]

[3] The proposal doesn't recommend any version of "git diff" with a
single <tree> argument. Fortunately we can continue to provide
compatible support for both such existing uses since they differ based
on the presence or absence of the --cached option.

Anyway, that's a proposal for some diff commands if we had the
opportunity to do it from scratch. I'm not a trained user of git that
would be impacted by this change, so I can't make any fair comment on
whether the change would be worth making or not. But I would
definitely be interested in hearing what existing users of git think
of the idea.

And of course, I am glad to fix up the implementation and all the
documentation as necessary to implement this proposal if people think
it's a good idea.

-Carl

[*] It's not the original topic of this post, but now that I've
finished this, I realize that if the diff proposal were implemented
then "commit-files" would make a dandy replacement for "commit
-a". That could lead to finally providing the parallel preview
commands I originally wanted:

	git diff-index	# as preview for
	git commit-index
and:
	git diff-files	# as preview for
	git commit-files

Then "git commit" would just be a shortcut for git commit-index.

(Oh, and that would also lead to a natural "git ci" abbreviation too,
if desired. This would parallel the "ci == checkin" abbreviation that
some other systems provide.)

I think the separate notions of commit-index and commit-files would do
a good job of allowing for simple tutorials, (eliminates the "what the
heck is -a all about?" questions), that also don't contribute to
general index-unawareness lead to later index-confusion as the current
"git diff; git commit -a" does.

This might even lead to a natural distinction between "git
status-index" and "git status files" too.

On Wed, 8 Feb 2006, Carl Worth wrote:

I would be using this before every "got commit" to get a preview, and
it seems like a painfully long name for such a common operation. Even
its shortcut form:

	git diff --cached HEAD

is among the longest of the "git diff" shortcuts.

[ snip snip ]

Yeah, I agree. The "git diff" semantics are incomplete, and the choices 
may not be (aka "almost certainly aren't") the best. The recent "git show" 
actually came from a very common use (well, for me, at least) for 
"git-diff-tree" that the "git diff" command simply didn't support.

I don't think it's really worth changing the low-level command naming: 
nopefully few enough people use them that it doesn't matter, but that's 
also what scripts out there are likely to have, so the pain factor when 
something breaks is high enough.

But making "git diff" itself more often "Do The Right Thing(tm)" would 
probably be a good idea.

In particular, I think the real culprit is the plain "git diff" with no 
arguments at all. Right now it ends up showing just a piece of the 
picture, and the piece it shows is incomplete enough to be irritating.

So I would prefer a slightly different fix from yours: instead of changing 
the core commands (which hopefully people mainly use in scripts where 
stability is more important than "intuitive and easy defaults"), how about 
_just_ changing the meaning of "git diff"?

Right now a plain "git diff" only shows the differences in the current 
tree against the index. I think that was just the wrong choice. I think 
almost everybody would actually prefer the default to be to show the 
difference against HEAD.

The largest reason for why "git diff" defaults to diffing against the 
index (and not head) is _literally_ that it's faster. Yes, I'm ashamed, 
but "git diff HEAD" takes 0.35 seconds for me, and "git diff" takes 0.07
seconds, and largely based entirely on that (and not "what would users 
want"), I made the bad choice to default to git-diff-files for the "no 
arguments" case.

I hang my head in shame. I just like the _instantness_ of "nothing 
changed". That's a bad reason for choice of interfaces, though.

One option is to realize that "git status" is actually a much better thing 
to use (because it shows _both_ the "HEAD to index" _and_ "index to 
current tree" format).

Teaching "git status" to take a "-p" flag (for "patch" - or -v for 
"verbose") might actually be a good thing. Then, instead of "git diff", 
you'd use "git status -p" and it would show you what the differences are 
in the index, and what they are in the tree, so you'd _really_ know what 
"git commit" in all its glory would do.

		Linus

On Wed, 8 Feb 2006, Linus Torvalds wrote:

The largest reason for why "git diff" defaults to diffing against the 
index (and not head) is _literally_ that it's faster. Yes, I'm ashamed, 
but "git diff HEAD" takes 0.35 seconds for me, and "git diff" takes 0.07
seconds, and largely based entirely on that (and not "what would users 
want"), I made the bad choice to default to git-diff-files for the "no 
arguments" case.

I hang my head in shame. I just like the _instantness_ of "nothing 
changed". That's a bad reason for choice of interfaces, though.

Btw, it was kind of luck (but it's definitely true) that the current "git 
diff" semantics happen to also be what you do want during merges when you 
try to resolve a conflict (while "git diff HEAD" is much less useful).

So making "git diff" default to diffing against HEAD is actually the wrong 
thing to do during merging, where the current behaviour is exactly what 
you'd want.

So I dunno. 

		Linus

Reading bits an pieces of your email gives an idea:
[...given at the end of this email]
On Wed, Feb 08, 2006 at 04:29:44PM -0800, Carl Worth wrote:

So, here I am as a newly converted index-embracer---no more index
denying from me.

However, I'm still trying to wrap my brain around the various diff
commands that git provides and how they would fit into my workflow,
Junio and I have touched on this already in a previous thread, but I'm
starting here with more fresh and complete analysis of the UI around
diff.

...

[*] It's not the original topic of this post, but now that I've
finished this, I realize that if the diff proposal were implemented
then "commit-files" would make a dandy replacement for "commit
-a". That could lead to finally providing the parallel preview
commands I originally wanted:

	git diff-index	# as preview for
	git commit-index
and:
	git diff-files	# as preview for
	git commit-files

Then "git commit" would just be a shortcut for git commit-index.

(Oh, and that would also lead to a natural "git ci" abbreviation too,
if desired. This would parallel the "ci == checkin" abbreviation that
some other systems provide.)

I think the separate notions of commit-index and commit-files would do
a good job of allowing for simple tutorials, (eliminates the "what the
heck is -a all about?" questions), that also don't contribute to
general index-unawareness lead to later index-confusion as the current
"git diff; git commit -a" does.

This might even lead to a natural distinction between "git
status-index" and "git status files" too.

...and now the idea:
What if we leave the normal long commands alone, so scripts and
muscle memory are not disturbed, but create some short commands
that give intuitive true parallel previews:
  "git di" and "git ci" (diff or commit index)
  "git df" and "git cf" (diff or commit files)
  "git dm" and "git cm" (diff or commit merge)
and of course any other little bits as needed like:
  "git si", "git sf", etc. (status index, status file, etc.)
This would give a chance to "start over" with clear commands,
without causing a ruckus among existing loyal users.  WDYT?

--Tim Larson

On Wed, 8 Feb 2006 17:21:49 -0800 (PST), Linus Torvalds wrote:

quoted

The largest reason for why "git diff" defaults to diffing against the 
index (and not head) is _literally_ that it's faster.

That's certainly surprising history to learn.

Btw, it was kind of luck (but it's definitely true) that the current "git 
diff" semantics happen to also be what you do want during merges when you 
try to resolve a conflict (while "git diff HEAD" is much less useful).

So making "git diff" default to diffing against HEAD is actually the wrong 
thing to do during merging, where the current behaviour is exactly what 
you'd want.

I discovered this during the analysis and guess (incorrectly) that it
was the motivation for the behavior of "git diff". Initially I think I
would have made a proposal that had "git diff" creating a diff from
HEAD except that I discovered two different targets that would be
useful when diffing from HEAD:

	HEAD -> index
and:	HEAD -> files

so I chose "index" and "files" as the naming distinction there. In
light of those, I couldn't think of any good naming for the remaining:

	index -> files

operation, so it seemed just as well to leave it with a bare "diff" as
before.

But, as you've pointed out, my trying to create new high-level
diff-index and diff-files operations fails in that it clashes with the
existing low-level uses of those names, (which scripts may already be
depending on).

I totally neglected to consider the pain of migrating scripts.

I still think the three above operations are the most commonly needed,
but I don't yet have good suggestions for names for them.

I think I'll be back with some new proposals after considering your
suggestion to expand git status.

-Carl