Re: [PATCHv3 3/9] pack-objects: Allow --max-pack-size to be used together with --stdout

On Monday 16 May 2011, Shawn Pearce wrote:

On Sun, May 15, 2011 at 14:37, Johan Herland [off-list ref] wrote:

quoted

Currently we refuse combining --max-pack-size with --stdout since
there's no way to make multiple packs when the pack is written to
stdout. However, we want to be able to limit the maximum size of the
pack created by --stdout (and abort pack-objects if we are unable to
meet that limit).

Therefore, when used together with --stdout, we reinterpret
--max-pack-size to indicate the maximum pack size which - if exceeded
- will cause pack-objects to abort with an error message.

...

quoted

               if (pack_to_stdout) {
+                       if (nr_written != nr_remaining)
+                               die("unable to make pack within the pack size"
+                                   " limit (%lu bytes)", pack_size_limit);

I think this is too late. We have already output a bunch of data, up
to the size limit at this point. If the size limit is non-trivial
(e.g. 5 MB) we have already sent most of that to the remote side, and
its already written some of that out to disk.

I'd like this to be a soft limit derived from the reused object sizes.
When planning the pack by looking at where we will reuse an object
from, sum those sizes. If the sum of these sizes would break this
limit, then we abort before even writing the pack header out.

I agree, but it's currently late Sunday (early Monday), and after
looking at this for a while, I'm no longer thinking straight.
If someone that groks the pack-objects internal could help out, I'd be
really grateful. AFAICS, we need to drill into prepare_pack() to find
the details needed to estimate the total pack size, but I don't know
exactly which data structure(s) holds the data needed. We probably need
to accumulate a pack size estimate in find_deltas(), and then sum those
across the threads, before we finally compare the total estimate to
pack_size_limit prior to calling write_pack_file().


...Johan

-- 
Johan Herland, [off-list ref]
www.herland.net