[RFH] zlib gurus out there?

I've been staring at reusing existing data while packing, and
this occurred to me...

During packing, suppose that we chose to store an object in
base form, undeltified.  And also suppose we have that object
loose in .git/objects/??/ directory.  We already have it in
deflated form, but with its own header.  I started wondering if
we can somehow reuse this.

A short object format brush-up lesson is in order here.  

* An undeltified object in a pack is represented like this:

 (1) the header is a dense variable size binary data, that
     encodes type and inflated length;
 (2) deflated data immediately follows the header.

* On the other hand, a loose object is represented like this:

 (1) the header looks like sprintf("%s %lu%c", type, len, 0);
 (2) concatenate the data to the header;
 (3) SHA1 checksum of the above becomes the object name.
 (4) deflate the header and data using the same z_stream, in two
     steps, like this (sha1_file.c::write_sha1_file):

	/* Compress it */
	stream.next_out = compressed;
	stream.avail_out = size;

	/* First header.. */
	stream.next_in = hdr;
	stream.avail_in = hdrlen;
	while (deflate(&stream, 0) == Z_OK)
		/* nothing */;

	/* Then the data itself.. */
	stream.next_in = buf;
	stream.avail_in = len;
	while (deflate(&stream, Z_FINISH) == Z_OK)
		/* nothing */;
	deflateEnd(&stream);
	size = stream.total_out;

So I thought... if we cause a full flush after the header part,
I can find the flush boundaries from a loose object file and
copy the rest into a packfile I am generating, after placing the
binary encoded header.  If this works, we do not have to inflate
loose object to read it and deflate it to store that in the
pack.  We will get a better packing as well, since we deflate
loose objects with Z_BEST_COMPRESSION, while packs are done with
Z_DEFAULT_COMPRESSION.  While pack-objects read from a loose
object, if we can detect that there is no full flush after the
header, we would do the traditional inflate-deflate cycle, so
this would be backward compatible.

However, I am stuck with the first step, which is to do a full
flush after the header.  An obvious change to the code quoted
above writes out a corrupt object:

	/* First header.. */
	stream.next_in = hdr;
	stream.avail_in = hdrlen;
-	while (deflate(&stream, 0) == Z_OK)
+	while (deflate(&stream, Z_FULL_FLUSH) == Z_OK)
		/* nothing */;

git-fsck-objects complains that sha1 does not match.  It appears
that the sha1_file.c::unpack_sha1_rest() somehow barfs upon
seeing the full flush, but I haven't dug into it yet.

Would anybody with more experience with zlib want to help?

On Tue, 7 Mar 2006, Junio C Hamano wrote:

However, I am stuck with the first step, which is to do a full
flush after the header.  An obvious change to the code quoted
above writes out a corrupt object:

	/* First header.. */
	stream.next_in = hdr;
	stream.avail_in = hdrlen;
-	while (deflate(&stream, 0) == Z_OK)
+	while (deflate(&stream, Z_FULL_FLUSH) == Z_OK)
		/* nothing */;

No, I don't think that's good. You're only doing a partial deflate, you 
can't ask for a Z_FULL_FLUSH. That only works if you give it the whole 
buffer, and you don't.

iirc, wtf, wdik, and ianal.

		Linus

Data after Z_FULL_FLUSH will be compressed independently of the
header, and could therefore be reused without recompressing when
creating a pack.

---

This passes "make test" and unpacking of the whole git repo with
git-fsck-objects afterwards.

However, a straight reuse still will not be possible, because
sha1write_compressed() uses deflateInit(&stream, Z_DEFAULT_COMPRESSION),
which writes zlib headers around the deflate stream, and the zlib footer
contains adler32 checksum.  So, as a minimum, you will need to
decompress the object data, calculate its adler32 checksum and write the
zlib header yourself.

 sha1_file.c |    7 ++++++-
 1 files changed, 6 insertions(+), 1 deletions(-)

8b12d9a58e87a4c5b5a2a7b20d06fe29a5afb903

diff --git a/sha1_file.c b/sha1_file.c
index a80d849..34d4da4 100644
--- a/sha1_file.c
+++ b/sha1_file.c

@@ -1399,7 +1399,8 @@ int write_sha1_file(void *buf, unsigned 
 	/* Set it up */
 	memset(&stream, 0, sizeof(stream));
 	deflateInit(&stream, Z_BEST_COMPRESSION);
-	size = deflateBound(&stream, len+hdrlen);
+	/* Additional 6 bytes for the Z_FULL_FLUSH marker */
+	size = deflateBound(&stream, hdrlen) + 6 + deflateBound(&stream, len);
 	compressed = xmalloc(size);
 
 	/* Compress it */

@@ -1412,6 +1413,10 @@ int write_sha1_file(void *buf, unsigned 
 	while (deflate(&stream, 0) == Z_OK)
 		/* nothing */;
 
+	/* Flush before data */
+	while (deflate(&stream, Z_FULL_FLUSH) == Z_OK)
+		/* nothing */;
+
 	/* Then the data itself.. */
 	stream.next_in = buf;
 	stream.avail_in = len;

-- 
1.2.GIT