On Thu, May 08, 2025 at 03:47:31PM -0400, Jeff King wrote:

On Thu, May 08, 2025 at 08:47:47PM +0200, Michal Suchánek wrote:

quoted

If you have one of those filesystems that support deduplication on
filesystem level you could make each snapshot as a full repository with
all objects unpacked, and the filesystem would deduplicate the objects
for you.

The downside is that you have no way to do multiple full backups this
way, and you would have to use something else for that (such as those
bundles, or plain archiving the repository as files in a tar archive or
such.

This is tempting, but I suspect that storing the objects unpacked will
become unfeasibly large, because you are missing out on delta
compression in the packfiles. You can compare the on-disk and
uncompressed sizes of objects in a repo like this:

  git cat-file --batch-all-objects --unordered \
               --batch-check='%(objectsize:disk) %(objectsize)' |
  perl -alne '
    $disk += $F[0];
    $true += $F[1];
    END {
      print "$true / $disk = ", int($true / $disk);
    }
  '

It's not entirely fair because the "true" size is missing out on zlib
compression that loose objects would get. But that's at best going to be
about 4:1 (and in practice worse, since trees are full of sha1 hashes
that don't compress very well).

In my copy of linux.git, that yields ~135G versus ~2.4G, for a factor of
56. Even if we grant 4:1 compression from zlib, that's still inflating
your on-disk repository by a factor of 14.

So with this estimate you recoup that size inflation after 14
incremental backups.

Since no other working incremantal backup strategy was proposed so far
this is the best one ;-)

Thanks

Michal