Re: [PATCH v2] rev-list --disk-usage

Jeff King [off-list ref] writes:

But in practice, we've found this kind of naive --disk-usage useful for
answering questions like:

  - do I need all of these objects? Comparing "rev-list --disk-usage
    --objects --all", "rev-list --disk-usage --objects --all --reflog",
    and "du objects/pack/*.pack" will tell you if a prune/repack might
    help, and whether expiring reflogs makes a difference.

  - the size of the shared alternates repo for a set of forks has
    jumped. Comparing "rev-list --disk-usage --objects --remotes=$base
    --not --remotes=$fork" will tell you what's reachable from a fork
    but not from the base (we use "refs/remotes/$id/*" to keep track of
    fork refs in our alternates repo). This can be junk like somebody
    forking git/git and then uploading a bunch of pirated video files.
    :)

  - likewise, the size of cloning a single repo may jump. Comparing
    "rev-list --disk-usage --objects HEAD..$branch" for each branch
    might show that one branch is an outlier (e.g., because somebody
    accidentally committed a bunch of build artifacts).

In those kinds of cases, it's not usually "oh, this version is twice as
big as this other one". It's more like "wow, this branch is 100x as big
as the other branches", and little decisions like delta direction are
just noise. I imagine that in those cases the uncompressed object sizes
would probably produce similar patterns and answers. But it's actually
faster to produce the on-disk sizes. :)

Thanks.

I kind of feel sad to have a nice write-up like this only in the
list archive.  Is there a section in our documentation set to keep
collection of such a real-life use cases?  Perhaps the examples
section of manpages is the closest thing, but it looks a bit too
narrowly scoped for the example section of "rev-list" manpage.

THanks.

On Wed, Feb 10, 2021 at 08:31:08AM -0800, Junio C Hamano wrote:

Jeff King [off-list ref] writes:

quoted

But in practice, we've found this kind of naive --disk-usage useful for
answering questions like:

[...]

I kind of feel sad to have a nice write-up like this only in the
list archive.  Is there a section in our documentation set to keep
collection of such a real-life use cases?  Perhaps the examples
section of manpages is the closest thing, but it looks a bit too
narrowly scoped for the example section of "rev-list" manpage.

Agreed on both counts. If this gets put into a release, I suspect Taylor
would cover it in a release blog post. That is not quite the same thing
as having it in the documentation, but it may provide more search engine
boost than the list archive. I dunno.

-Peff

On Wed, Feb 10, 2021 at 03:38:58PM -0500, Jeff King wrote:

On Wed, Feb 10, 2021 at 08:31:08AM -0800, Junio C Hamano wrote:

quoted

Jeff King [off-list ref] writes:

quoted

But in practice, we've found this kind of naive --disk-usage useful for
answering questions like:

[...]

I kind of feel sad to have a nice write-up like this only in the
list archive.  Is there a section in our documentation set to keep
collection of such a real-life use cases?  Perhaps the examples
section of manpages is the closest thing, but it looks a bit too
narrowly scoped for the example section of "rev-list" manpage.

Agreed on both counts. If this gets put into a release, I suspect Taylor
would cover it in a release blog post. That is not quite the same thing
as having it in the documentation, but it may provide more search engine
boost than the list archive. I dunno.

Yeah, this is the perfect sort of thing for those blog posts.

But it makes sense to include some of these examples in our own
documentation here, too. git-rev-list(1) doesn't have an EXAMPLES
section, but maybe it should.

-Peff

Thanks,
Taylor

On Wed, Feb 10, 2021 at 06:15:16PM -0500, Taylor Blau wrote:

quoted

quoted

I kind of feel sad to have a nice write-up like this only in the
list archive.  Is there a section in our documentation set to keep
collection of such a real-life use cases?  Perhaps the examples
section of manpages is the closest thing, but it looks a bit too
narrowly scoped for the example section of "rev-list" manpage.

Agreed on both counts. If this gets put into a release, I suspect Taylor
would cover it in a release blog post. That is not quite the same thing
as having it in the documentation, but it may provide more search engine
boost than the list archive. I dunno.

Yeah, this is the perfect sort of thing for those blog posts.

But it makes sense to include some of these examples in our own
documentation here, too. git-rev-list(1) doesn't have an EXAMPLES
section, but maybe it should.

I think this is the "narrowly scoped" bit from Junio's response above.
It would be a bit weird to have an examples section for rev-list that
only mentions this rather obscure feature.

-Peff

On Thu, Feb 11 2021, Jeff King wrote:

On Wed, Feb 10, 2021 at 06:15:16PM -0500, Taylor Blau wrote:

quoted

quoted

quoted

I kind of feel sad to have a nice write-up like this only in the
list archive.  Is there a section in our documentation set to keep
collection of such a real-life use cases?  Perhaps the examples
section of manpages is the closest thing, but it looks a bit too
narrowly scoped for the example section of "rev-list" manpage.

Agreed on both counts. If this gets put into a release, I suspect Taylor
would cover it in a release blog post. That is not quite the same thing
as having it in the documentation, but it may provide more search engine
boost than the list archive. I dunno.

Yeah, this is the perfect sort of thing for those blog posts.

But it makes sense to include some of these examples in our own
documentation here, too. git-rev-list(1) doesn't have an EXAMPLES
section, but maybe it should.

I think this is the "narrowly scoped" bit from Junio's response above.
It would be a bit weird to have an examples section for rev-list that
only mentions this rather obscure feature.

I don't think the lack of an EXAMPLES section or the relative obscurity
of the switch should preclude us from adding useful documentation.

Yes it would feel a bit out of place, but we can always have a
sub-section of EXAMPLES, and we've got to start somewhere.

In this case I don't see why it couldn't be added to OPTIONS, we've got
some very long discussion there already, and as long as there's a clear
separation in prose from an initial brief discussion of the switch and
further prose it won't be confusing for readers, they can just page past
the details.

On Thu, Feb 11, 2021 at 01:04:26PM +0100, Ævar Arnfjörð Bjarmason wrote:

quoted

I think this is the "narrowly scoped" bit from Junio's response above.
It would be a bit weird to have an examples section for rev-list that
only mentions this rather obscure feature.

I don't think the lack of an EXAMPLES section or the relative obscurity
of the switch should preclude us from adding useful documentation.

Yes it would feel a bit out of place, but we can always have a
sub-section of EXAMPLES, and we've got to start somewhere.

Fair enough. Here are some patches (to go on top of jk/rev-list-disk-usage,
though obviously the first one could be applied independently).

In this case I don't see why it couldn't be added to OPTIONS, we've got
some very long discussion there already, and as long as there's a clear
separation in prose from an initial brief discussion of the switch and
further prose it won't be confusing for readers, they can just page past
the details.

It's already big and scary enough that I prefer starting an EXAMPLES
section. :)

By the way, there's one other finishing touch we might consider:
enabling --use-bitmap-index automatically when bitmaps are present, for
requests that produce the identical answer (so _not_ a regular
traversal, because the output order and presence of pathnames are
different there). I'd prefer to do that as a separate series, though,
since there are multiple arguments that might benefit (like --count).

  [1/2]: docs/rev-list: add an examples section
  [2/2]: docs/rev-list: add some examples of --disk-usage

 Documentation/git-rev-list.txt | 93 ++++++++++++++++++++++++++++++++++
 1 file changed, 93 insertions(+)

-Peff

We currently don't show any examples of using git-rev-list at all. Let's
add some pretty elementary examples. They likely seem obvious to anybody
who has worked with the tool for a while, but my purpose here is
two-fold:

  - they may be enlightening to people who haven't used the tool a lot
    to give a general flavor of how it is meant to be used

  - they can serve as a starting point for adding more interesting
    examples (we can do that without the basic ones, of course, but I
    think it makes sense to show off the building blocks)

This set is far from exhaustive, but again, the purpose is to be a
starting point for further additions.

Signed-off-by: Jeff King <redacted>
---
I'm open to feedback on these. But please, if you have suggestions for
adding more, do it in the form of a patch on top. :)

 Documentation/git-rev-list.txt | 52 ++++++++++++++++++++++++++++++++++
 1 file changed, 52 insertions(+)

diff --git a/Documentation/git-rev-list.txt b/Documentation/git-rev-list.txt
index 5da66232dc..d7ff519b90 100644
--- a/Documentation/git-rev-list.txt
+++ b/Documentation/git-rev-list.txt

@@ -31,6 +31,58 @@ include::rev-list-options.txt[]
 
 include::pretty-formats.txt[]
 
+EXAMPLES
+--------
+
+* Print the list of commits reachable from the current branch.
++
+----------
+git rev-list HEAD
+----------
+
+* Print the list of commits on this branch, but not present in the
+  upstream branch.
++
+----------
+git rev-list @{upstream}..HEAD
+----------
+
+* Format commits with their author and commit message (see also the
+  porcelain linkgit:git-log[1]).
++
+----------
+git rev-list --format=medium HEAD
+----------
+
+* Format commits along with their diffs (see also the porcelain
+  linkgit:git-log[1], which can do this in a single process).
++
+----------
+git rev-list HEAD |
+git diff-tree --stdin --format=medium -p
+----------
+
+* Print the list of commits on the current branch that touched any
+  file in the `Documentation` directory.
++
+----------
+git rev-list HEAD -- Documentation/
+----------
+
+* Print the list of commits authored by you in the past year, on
+  any branch, tag, or other ref.
++
+----------
+git rev-list --author=you@example.com --since=1.year.ago --all
+----------
+
+* Print the list of objects reachable from the current branch (i.e., all
+  commits and the blobs and trees they contain).
++
+----------
+git rev-list --objects HEAD
+----------
+
 GIT
 ---
 Part of the linkgit:git[1] suite

-- 
2.30.1.989.g5e01c2f281

It's not immediately obvious why --disk-usage might be a useful thing.
These examples show off a few of the real-world cases I've used it for.

Signed-off-by: Jeff King <redacted>
---
 Documentation/git-rev-list.txt | 41 ++++++++++++++++++++++++++++++++++
 1 file changed, 41 insertions(+)

diff --git a/Documentation/git-rev-list.txt b/Documentation/git-rev-list.txt
index d7ff519b90..20bb8e8217 100644
--- a/Documentation/git-rev-list.txt
+++ b/Documentation/git-rev-list.txt

@@ -83,6 +83,47 @@ git rev-list --author=you@example.com --since=1.year.ago --all
 git rev-list --objects HEAD
 ----------
 
+* Compare the disk size of all reachable objects, versus those
+  reachable from reflogs, versus the total packed size. This can tell
+  you whether running `git repack -ad` might reduce the repository size
+  (by dropping unreachable objects), and whether expiring reflogs might
+  help.
++
+----------
+# reachable objects
+git rev-list --disk-usage --objects --all
+# plus reflogs
+git rev-list --disk-usage --objects --all --reflog
+# total disk size used
+du -c .git/objects/pack/*.pack .git/objects/??/*
+# alternative to du: add up "size" and "size-pack" fields
+git count-objects -v
+----------
+
+* Report the disk size of each branch, not including objects used by the
+  current branch. This can find outliers that are contributing to a
+  bloated repository size (e.g., because somebody accidentally committed
+  large build artifacts).
++
+----------
+git for-each-ref --format='%(refname)' |
+while read branch
+do
+	size=$(git rev-list --disk-usage --objects HEAD..$branch)
+	echo "$size $branch"
+done |
+sort -n
+----------
+
+* Compare the on-disk size of branches in one group of refs, excluding
+  another. If you co-mingle objects from multiple remotes in a single
+  repository, this can show which remotes are contributing to the
+  repository size (taking the size of `origin` as a baseline).
++
+----------
+git rev-list --disk-usage --objects --remotes=$suspect --not --remotes=origin
+----------
+
 GIT
 ---
 Part of the linkgit:git[1] suite

-- 
2.30.1.989.g5e01c2f281

On Wed, Feb 17, 2021 at 06:31:07PM -0500, Jeff King wrote:

It's already big and scary enough that I prefer starting an EXAMPLES
section. :)

The patches you sent below are great. I think that it's easy to nitpick
and say "oh, you should have added this or that example, too", but I
think you gave a great set of starting examples.

I'd be happy to see this merged so that others can add more examples on
top.

By the way, there's one other finishing touch we might consider:
enabling --use-bitmap-index automatically when bitmaps are present, for
requests that produce the identical answer (so _not_ a regular
traversal, because the output order and presence of pathnames are
different there). I'd prefer to do that as a separate series, though,
since there are multiple arguments that might benefit (like --count).

This would be really neat. I look forward to it.

Thanks,
Taylor