Re: [PATCH v2] pack-objects: use streaming interface for reading large loose blobs

Nguyễn Thái Ngọc Duy [off-list ref] writes:

+	st = open_istream(sha1, &type, &sz, NULL);
+	if (!st)
+		die(_("unable to read %s"), sha1_to_hex(sha1));
+
+	memset(&stream, 0, sizeof(stream));
+	git_deflate_init(&stream, pack_compression_level);
+
+	if (type != OBJ_BLOB)
+		die("BUG: %s is not a blob", sha1_to_hex(sha1));

Just a comment, not an expression of preference, but if we are treating
istream interface as an add-on, I wonder it might make sense not to die in
these places, and instead fall back to the usual "read and then write"
codepath.

To put it another way, I am wondering if it makes the logic more clear if
you restructure the calling side a bit more, perhaps like this:

        if (!to_reuse) {
                if (!usable_delta) {
                        /* we know we will write as base object */
                        ... do *NOT* do read_sha1_file() here !!!
                } else if ... delta cases {
                        decide delta_data and z_delta_size as before
                }

                if (entry->z_delta_size) {
                        datalen = entry->z_delta_size;
                        hdrlen = encode_in_pack_object_header(type, size, header);
                        ... write either OFS_DELTA or REF_DELTA here
                } else {
                        /* base object case */
                        if (write_object_in_base_representation(f, sha1)) {
                                /* stream interface punted */
                                read_sha1_file();
                                do_compress();
                                write it;
                        }
                }
        } else { /* reuse case ... */

Two comments:

 - In get_object_details(), we do this:

		if (big_file_threshold <= entry->size)
			entry->no_try_delta = 1;

   The new code is inconsistent with respect to the boundary conditions,
   when the size is exactly at the threshold.

 - Magic condition "buf has NULL means we will read via streaming
   interface" feels somewhat hacky and tasteless.  I am borderline OK with
   such a hack whose scope is clearly limited to a short function like
   this one, but at least it needs to be documented.  I'd rather see the
   conditionals that look at !buf/buf are rewritten to use a new bool
   variable similar to to_reuse and usable_delta whose name tells the
   reader more explicitly what is going on, though.