From: "David S. Miller" <davem@davemloft.net> Date: 2016-06-15 22:42:00
To get a clean history to push to Linus, I typically blow
away my trees and make fresh ones to stick patches into
which I want to merge.
That mostly works fine here on my local systems, but I know this
brings the master.org mirroring system to it's knees. So what is the
generally condoned way to do stuff like this in a more friendly way?
Should I:
1) Do a git pull from Linus's tree once he takes my changes, then
ask GIT to prune the tree? How do I do that and how does it work?
2) Should I use .git/object/ database symlinking?
Are there any scripts out there which do this automatically?
Something as simple to run as "git-pull-script" and it takes
care of using links when possible on a local filesystem.
It takes sometimes an hour for my tree updates on master.kernel.org
to propagate to rsync.kernel.org so I can ask Linus to pull.
That's crazy.
From: Jeff Garzik <hidden> Date: 2016-06-15 22:42:00
David S. Miller wrote:
To get a clean history to push to Linus, I typically blow
away my trees and make fresh ones to stick patches into
which I want to merge.
That mostly works fine here on my local systems, but I know this
brings the master.org mirroring system to it's knees. So what is the
generally condoned way to do stuff like this in a more friendly way?
Should I:
1) Do a git pull from Linus's tree once he takes my changes, then
ask GIT to prune the tree? How do I do that and how does it work?
It takes sometimes an hour for my tree updates on master.kernel.org
to propagate to rsync.kernel.org so I can ask Linus to pull.
That's crazy.
Unfortunately you cannot fix this by changing your actions. This is the
cumulative effect of all the git kernel trees on kernel.org. It now
takes over an hour for my non-git changes to propagate from master to
the mirrors, as well.
This is all due to the rsync sweeps, which have to scan metric tons of
inodes and dentries. Orders of magnitude over the pre-git days.
ftpadmin@kernel.org folks are supposedly working on an inotify-based
system, and an improved rsync application. No ETA or details.
As an aside, cold-cache, git really punishes my disks. Ted T'so noted
that it really drains laptop batteries, too.
2) Should I use .git/object/ database symlinking?
Are there any scripts out there which do this automatically?
Something as simple to run as "git-pull-script" and it takes
care of using links when possible on a local filesystem.
On both kernel.org and locally, I use 'cp -al' to duplicate the initial
.git/objects directory, and then rsync (->kernel.org) or git-pull-script
(<-kernel.org) to update it after that.
That definitely helps.
Maybe somebody needs to script a relink cron job for kernel.org?
Jeff
This is all due to the rsync sweeps, which have to scan metric tons of
inodes and dentries. Orders of magnitude over the pre-git days.
Well, the real solution is to use a git-aware protocol, not rsync.
rsync is wonderful for prototyping, and I wanted to make the database
rsync'able for that reason, but it clearly doesn't scale.
I think I'll make a "pack"/"unpack" pair that just packs all the necessary
objects between two commits. Then you can basically sync the object file
by doing
git-pack OLD..NEW | ssh other-end git-unpack
and you'd basically be done. It looks pretty easy to do, too..
Linus
From: Jeff Garzik <hidden> Date: 2016-06-15 22:42:00
Linus Torvalds wrote:
On Sat, 25 Jun 2005, Jeff Garzik wrote:
quoted
This is all due to the rsync sweeps, which have to scan metric tons of
inodes and dentries. Orders of magnitude over the pre-git days.
Well, the real solution is to use a git-aware protocol, not rsync.
rsync is wonderful for prototyping, and I wanted to make the database
rsync'able for that reason, but it clearly doesn't scale.
I think I'll make a "pack"/"unpack" pair that just packs all the necessary
objects between two commits. Then you can basically sync the object file
by doing
git-pack OLD..NEW | ssh other-end git-unpack
and you'd basically be done. It looks pretty easy to do, too..
The problem is kernel.org mirroring, not individual pushes and pulls,
really.
Would git-pack be the best solution for mirroring a bunch of git trees?
Jeff
The problem is kernel.org mirroring, not individual pushes and pulls,
really.
Would git-pack be the best solution for mirroring a bunch of git trees?
No guarantees, but here's a rough plan:
- I just committed a fairly trivial change to add a "--objects" flag to
git-rev-list, which allows you to basically say "I want to see the
difference not just in commit ID's, but also trees and blobs"
What does that mean? It means that in a mirroring schenario, you can,
for each git tree, do:
(a) On the slave:
cat .git/refs/*/* | sort | uniq > slave-ref-list
(b) On the master:
cat .git/refs/*/* | sort | uniq > master-ref-list
(c) On the master:
cmp $master-ref-list $slave-ref-list && exit 1
list=$(cat master-ref-list)
for i in $(cat slave-ref-list)
do
list=$list ^$i
done
git-rev-list --objects $list
and now that "git-rev-list" will list every object that needs to be
copied from the master to the slave. No need to read huge directories
etc, you get the list computed for you.
yeah, it clearly needs some refining to be useful, but I think you can
kind of see how it would work.
Now, the secondary advantage of this is that once you don't use rsync as
the mirroring method, you can now change the filesystem object database
layout. In particular, the packing thing that Chris Mason was working on
at some point suddenly becomes a lot more viable.
(In fact, more than that. You can make a single packed blob for all
"historical" objects, and that also gives you an efficient archive format
- if you're not required to have the full filesystem layout, you could
have a much more efficient packing that you basically do once a week or
something, so that you only keep the last week in the regular "one file
per object" format).
Linus
yeah, it clearly needs some refining to be useful, but I think you can
kind of see how it would work.
Ok, here's how it works.
- Pick a starting commit (or a hundred)
- Pick an ending commit (or a hundred)
- generate the list of objects in between them
git-rev-list --object end ^start > object-list
- Pack that list of objects into an "object pack":
git-pack-objects out < object-list
(This actually generates two files: "out.idx" is the index file,
"out.pack" is the data file, but I'll make it concatenate the two at
some point)
- move the pack-files over somewhere else
- unpack them
git-unpack-objects out
and you're done.
Now, the reason I use "pack" and "unpack" instead of just "tar" to
transport the objects is that this allows me to do a fairly efficient
packing. I wanted these pack-files to be independent (ie they do _not_
depend on any objects outside of the pack-file), but within the objects
described in the pack I cna do delta-compression.
Now, that doesn't much help for small updates (where the objects are just
unrelated and have no deltas), but it helps increasingly for big ones. The
biggest one obviously being the whole path from the start to the HEAD..
For example, the "du -sh .git/objects" for the git project itself is 17MB
for me, and I can do:
torvalds@ppc970:~/git> du -sh .git/objects
17M .git/objects
torvalds@ppc970:~/git> time git-rev-list --objects HEAD | git-pack-objects out
Packing 3656 objects
real 0m3.779s
user 0m3.169s
sys 0m0.602s
torvalds@ppc970:~/git> ls -lh out.*
-rw-rw-r-- 1 torvalds torvalds 87K Jun 26 09:12 out.idx
-rw-rw-r-- 1 torvalds torvalds 2.0M Jun 26 09:12 out.pack
ie it packs down to a nice 2MB pack-file with a small index. Move that
over to somewhere else, and unpack it, and you'll get all the regular
objects (it doesn't move tags and refs over, you'll have to do that
outside of the packing).
Now, you can trade off some packing time to get a better pack:
torvalds@ppc970:~/git> time git-rev-list --objects HEAD | git-pack-objects --window=100 out
Packing 3656 objects
real 0m11.953s
user 0m11.294s
sys 0m0.663s
torvalds@ppc970:~/git> ls -lh out.*
-rw-rw-r-- 1 torvalds torvalds 87K Jun 26 09:14 out.idx
-rw-rw-r-- 1 torvalds torvalds 1.6M Jun 26 09:14 out.pack
and if you want to allow deep delta chains (the default delta depth
limiting is 10), you can get even better results:
torvalds@ppc970:~/git> time git-rev-list --objects HEAD | git-pack-objects --window=100 --depth=100 out
Packing 3656 objects
real 0m12.374s
user 0m11.704s
sys 0m0.659s
torvalds@ppc970:~/git> ls -lh out.*
-rw-rw-r-- 1 torvalds torvalds 87K Jun 26 09:16 out.idx
-rw-rw-r-- 1 torvalds torvalds 1.3M Jun 26 09:16 out.pack
but then unpacking will slightly heavier.
(Doing the same for the kernel is obviously much more expensive just
because the kernel is so much bigger. A big delta discovery window like
100 takes about fifteen minutes to pack on my machine, but gets the
current kernel archive down to 70MB or so. That's ok for a monthly "pack
all the objects" to keep size requirements down, but you clearly don't
want to do this all the time ;).
Now, perhaps the more interesting part is that I also designed the pack
format so that it should be a good "history" format, not just a way to
move objects from one place to the other. Ie if you worry about diskspace,
you can pack everything up to the now into one big pack, and then remove
the original objects.
Don't do that yet, btw - I haven't actually written the code to read stuff
out of packs if we don't find it in the object directory yet, but the
layout is such that it should be straightforward and pretty efficient (but
there a deep delta chain obviously _will_ cause a performance hit).
I actually like this approach better than having delta-objects in the
filesystem. Partly because the pack-file is self-contained, partly because
it also solves the fs blocking issue, yet is still efficient to look up
the results without having hardlinks etc to duplicate objects virtually.
And when you do the packing by hand as an "archival" mechanism, it also
doesn't have any of the downsides that Chris' packing approach had.
Nico? Chris? Interested in giving it a look? It's kind of a combination of
your things, generalized and then made to have fast lookup with the index.
Fast lookup doesn't matter for a normal unpack, of course, and if I just
always wanted to unpack all the objects (ie just an object transfer
mechanism) I'd have made the index be a toposort of the objects. But
because I wanted to be able to use it as an archival format, I needed it
to be "random-access" by object name. So the index is in fact a binary
tree (well, sorted array, so the lookup degenerates into a binary search)
with a top-level index splitting up the contents based on the first byte
(the same way the filesystem layout does).
Linus
From: Chris Mason <hidden> Date: 2016-06-15 22:42:00
On Sunday 26 June 2005 12:41, Linus Torvalds wrote:
On Fri, 24 Jun 2005, Linus Torvalds wrote:
quoted
yeah, it clearly needs some refining to be useful, but I think you can
kind of see how it would work.
Ok, here's how it works.
- Pick a starting commit (or a hundred)
- Pick an ending commit (or a hundred)
- generate the list of objects in between them
git-rev-list --object end ^start > object-list
- Pack that list of objects into an "object pack":
git-pack-objects out < object-list
Without having read the code, the big thing that hurt performance in my early
packed file work was compressing the whole packed file instead of individual
sub-objects. It takes more room to compress each object, but when I
compressed the whole thing read performance was quite bad.
-chris
From: Chris Mason <hidden> Date: 2016-06-15 22:42:00
On Sunday 26 June 2005 16:52, Chris Mason wrote:
quoted
git-rev-list --object end ^start > object-list
- Pack that list of objects into an "object pack":
git-pack-objects out < object-list
Without having read the code, the big thing that hurt performance in my
early packed file work was compressing the whole packed file instead of
individual sub-objects. It takes more room to compress each object, but
when I compressed the whole thing read performance was quite bad.
Sorry, fat fingered the send key...
The hard links were the biggest problem with my packed file patches, I think
the dynamic lookup in a separate packed file index is the best way to go.
-chris
Without having read the code, the big thing that hurt performance in my early
packed file work was compressing the whole packed file instead of individual
sub-objects. It takes more room to compress each object, but when I
compressed the whole thing read performance was quite bad.
Since I wanted random-access, compressing the whole thing just wasn't an
option.
Besides, the big space savings come from finding deltas, which is
obviously also a compression, just at a higher level. The biggest problem
there is to find a guess of objects to try to delta against, and right now
that part is pretty stupid and could possibly be improved (it just sorts
objects by size and tries to delta against "close" objects).
To generate a better sort _would_ actually be pretty close to doing a
global compression (it really does boil down to the same thing: finding
big sub-sequences, except it's in a "fragmented" space), but one issue is
that I don't want to read in the whole data set in one go, so it would
have to be based on some rolling hash or something. Davide pointed to
rzip, and a variation of that (which knows about object boundaries) might
work.
(You can also sort by filename, if you want to try. I don't track
filenames at all there and it's actually non-trivial to do, so that would
require some new and pretty nasty code, but it's possible in _theory_ at
least.)
Anyway, that's all potential improvement for generating better packing,
and it should certainly be possible without changing the format - just
generate a better initial sort, in otder to find more deltas (or rather,
find them faster by using a smaller window size).
So the stupid sort I have now does actually work, but exactly because it's
so stupid it wants a big window for best packing (because there might be a
lot of objects that aren't interesting), which in turn is quite expensive.
So a better sort would make a smaller window more effective.
[ Some numbers: a window of 10 objects is the default, and packs the
current kernel down to 77MB in 2m21s. A window of 20 objects improves
that packing to 71MB, but makes the packing time go up to 3m36s for me.
And a window of 100 gets us down to 62M but takes 11m54s.
A window of 200 (with a delta depth of 200 too - likely _way_ too deep
for normal use) gives you a 59M pack, but takes 20m59s, so there's
definitely a point of diminishing returns.
This is all for the current HEAD, which takes up 264M the "traditional"
git way and takes 141M without any deltas, just packed tightly with no
filesystem blocking.
Now, as you can notice that's actually a slightly sub-linear increase in
time, because as we find a delta, we will only accept smaller deltas in
the future, so we can often stop comparing even before we've reached the
maximum window size, and so effort is slightly less than linear because
there's effectively a constant component to part of it.
Also, the good news is that you probably don't want to generate one
humungous pack archive anyway, but you're likely better off doing a new
incremental pack every few months. So we'll never have the situation
that creating a pack gets increasingly more costly, since at some point
you just say "ok, I created a perfect pack for the first 4 months of
development, I'll now do subsequent packs on top of that instead".
The other good news is that a pack is also a natural boundary for fsck
(as in "ok, I found that object in a pack, so I won't bother going
deeper in the reachability chain"), so if you start packing your
repository, fsck will only have to worry about the objects that are
unpacked. That makes them work really naturally for archiving, ie this
all means that you can avoid a lot of overhead by packing your history
every once in a while, with it all being entirely transparent.
In other words, if you just pack every month, you can basically
guarantee that fsck costs etc never really go up, and your diskspace
also goes up only very slowly. The packed format is quite efficient in
many ways, but it is totally immutable (ie you can't add anything to an
archive - a pack stays the way it always was, and if you want to pack
more you have to either re-do the pack or just create a new one) ]
Linus
(You can also sort by filename, if you want to try. I don't track
filenames at all there and it's actually non-trivial to do, so that would
require some new and pretty nasty code, but it's possible in _theory_ at
least.)
Heh. It's actually very easy if you don't take the "name" too seriously,
and you just pick some random one, namely the first one that was used to
reach the entry.
Then, you might sort the objects on a hash based on the name, and get
tons of cheap deltas close-by.
It is _uglee_, but hey, it's a heuristic, and it happens to work pretty
well. It brought the kernel pack down to 59M even with just a small window
of 10.
Thanks to Davide for making me think about this hack, although he talked
about something much more proper (and harder) than this quick and
ugly heuristic ;)
The main change is that "git-rev-list --objects" has been changed to show
the name the object was reached through.
Linus
From: Nicolas Pitre <hidden> Date: 2016-06-15 22:42:00
On Sun, 26 Jun 2005, Linus Torvalds wrote:
On Fri, 24 Jun 2005, Linus Torvalds wrote:
quoted
yeah, it clearly needs some refining to be useful, but I think you can
kind of see how it would work.
Ok, here's how it works.
[...]
Nico? Chris? Interested in giving it a look? It's kind of a combination of
your things, generalized and then made to have fast lookup with the index.
Wow !
Look away for a few days and when you look back your own stuff has been
yanked out!
But actually I like this packing thing. Certainly much nicer to work
with and also more useful. And, since the pack file has its own
checksum it is possible to use that pack format (without the index
(which can be recreated from the pack file alone anyway) for network
transfer.
Here's one improvement to the pack format, breaking it early so it won't
affect anyone at this point: compressed object header. Instead of the
fixed 5 byte header, this patch convert it to a variable size granted
most object are small enough to save on the storage of the significant
size bytes which will be zero and packing the non-zero byte position
with the object type.
This can save up to 2% or 21KB on the test I
performed with the git repository.
@@ -120,18 +129,12 @@ static int unpack_non_delta_entry(structinflateEnd(&stream);if((st!=Z_STREAM_END)||stream.total_out!=size)gotoerr_finish;-switch(kind){-case'C':type_s="commit";break;-case'T':type_s="tree";break;-case'B':type_s="blob";break;-default:gotoerr_finish;-}-if(write_sha1_file(buffer,size,type_s,sha1)<0)+if(write_sha1_file(buffer,size,type,sha1)<0)die("failed to write %s (%s)",-sha1_to_hex(entry->sha1),type_s);-printf("%s %s\n",sha1_to_hex(sha1),type_s);+sha1_to_hex(entry->sha1),type);+printf("%s %s\n",sha1_to_hex(sha1),type);if(memcmp(sha1,entry->sha1,20))-die("resulting %s have wrong SHA1",type_s);+die("resulting %s have wrong SHA1",type);finish:st=0;
@@ -183,15 +186,13 @@ static int unpack_delta_entry(struct pacdie("truncated pack file");data=base_sha1+20;data_size=left-20;-printf("%s D %lu",sha1_to_hex(entry->sha1),delta_size);+printf("%s delta %lu",sha1_to_hex(entry->sha1),delta_size);printf(" %s\n",sha1_to_hex(base_sha1));if(dry_run)return0;-/* pack+5 is the base sha1, unless we have it, we need to-*unpackitfirst.-*/+/* unless we have the base sha1, we need to unpack it first. */if(!has_sha1_file(base_sha1)){structpack_entry*base;if(!find_pack_entry(base_sha1,&base))
@@ -236,7 +237,9 @@ static int unpack_delta_entry(struct pacstaticvoidunpack_entry(structpack_entry*entry){unsignedlongoffset,size,left;-unsignedchar*pack;+unsignedchar*pack,sizebits;+enumobject_typetype;+inti;/* Have we done this one already due to deltas based on it? */if(lookup_object(entry->sha1))
Here's one improvement to the pack format, breaking it early so it won't
affect anyone at this point: compressed object header. Instead of the
fixed 5 byte header, this patch convert it to a variable size granted
most object are small enough to save on the storage of the significant
size bytes which will be zero and packing the non-zero byte position
with the object type.
Ok, this is against an older version and doesn't have that "read_sha1"
thing, but yes, something like this would work.
I'd prefer the encoding to be a bit different, though: make the size be
encoded in seven bits per byte, with the high bit meaning "more to come".
We can use four bits from the "type" byte for the initial value, making
lengths 0-15 be free.
unsigned long size;
unsigned char c;
c = *pack++;
type =
size = c & 15;
type = (c >> 4) & 7;
while (c & 0x80) {
c = *pack++;
size = (size << 7) + (pack & 0x7f);
}
or something. That's even denser.
However, I also end up wanting to add a "global header" to the pack-file,
that contains at least the number of objects packed. We may not know how
big the pack-file will be, but we'll at least know how many objects it
has before we start writing it.
Linus
From: Nicolas Pitre <hidden> Date: 2016-06-15 22:42:00
On Tue, 28 Jun 2005, Linus Torvalds wrote:
On Tue, 28 Jun 2005, Nicolas Pitre wrote:
quoted
Here's one improvement to the pack format, breaking it early so it won't
affect anyone at this point: compressed object header. Instead of the
fixed 5 byte header, this patch convert it to a variable size granted
most object are small enough to save on the storage of the significant
size bytes which will be zero and packing the non-zero byte position
with the object type.
Ok, this is against an older version and doesn't have that "read_sha1"
thing, but yes, something like this would work.
I'd prefer the encoding to be a bit different, though: make the size be
encoded in seven bits per byte, with the high bit meaning "more to come".
We can use four bits from the "type" byte for the initial value, making
lengths 0-15 be free.
unsigned long size;
unsigned char c;
c = *pack++;
type =
size = c & 15;
type = (c >> 4) & 7;
while (c & 0x80) {
c = *pack++;
size = (size << 7) + (pack & 0x7f);
}
or something. That's even denser.
OK. New patch below.
However, I also end up wanting to add a "global header" to the pack-file,
that contains at least the number of objects packed. We may not know how
big the pack-file will be, but we'll at least know how many objects it
has before we start writing it.
Probably a version signature would be a good thing too.
=====
This patch adds compressed object header. Instead of the fixed 5 byte
header, this patch convert it to a variable size granted most object are
small enough to save on the storage of the size's most significant bytes
which are zero, even ffolding bits with the object type. Objects up to
2047 bytes in size will have a header of only 2 bytes.
Signed-off-by: Nicolas Pitre <redacted>
@@ -63,21 +56,30 @@ static unsigned long write_object(structdie("object %s size inconsistency (%lu vs %lu)",sha1_to_hex(entry->sha1),size,entry->size);/*-*Theobjectheaderisabyteof'type'followedbyfourbytesof-*length,exceptfordeltasthathasthe20bytesofdeltasha-*instead.+*Theobjectheaderfirstbytehasitslow4bitsrepresentingthe+*objecttype,the4upperbitsindicatingwhichofthefollowing+*bytesareusedtobuildtheobjectsize.Fordeltaobjectsthe+*sha1ofthereferenceobjectisalsoappended.*/header[0]=entry->type;-hdrlen=5;if(entry->delta){-header[0]='D';-memcpy(header+5,entry->delta,20);+header[0]=PACKED_DELTA;buf=delta_against(buf,size,entry);size=entry->delta_size;-hdrlen=25;}-datalen=htonl(size);-memcpy(header+1,&datalen,4);+header[0]|=size<<3;+hdrlen=1;+datalen=size>>4;+while(datalen){+header[hdrlen-1]|=0x80;+header[hdrlen++]=datalen;+datalen>>=7;+}+if(entry->delta){+memcpy(header+hdrlen,entry->delta,20);+hdrlen+=20;+}+sha1write(f,header,hdrlen);datalen=sha1write_compressed(f,buf,size);free(buf);
@@ -168,13 +170,13 @@ static void check_object(struct object_eif(!sha1_object_info(entry->sha1,type,&entry->size)){if(!strcmp(type,"commit")){-entry->type='C';+entry->type=PACKED_COMMIT;}elseif(!strcmp(type,"tree")){-entry->type='T';+entry->type=PACKED_TREE;}elseif(!strcmp(type,"blob")){-entry->type='B';+entry->type=PACKED_BLOB;}elseif(!strcmp(type,"tag")){-entry->type='G';+entry->type=PACKED_TAG;}elsedie("unable to pack object %s of type %s",sha1_to_hex(entry->sha1),type);
@@ -120,22 +128,15 @@ static int unpack_non_delta_entry(structinflateEnd(&stream);if((st!=Z_STREAM_END)||stream.total_out!=size)gotoerr_finish;-switch(kind){-case'C':type_s="commit";break;-case'T':type_s="tree";break;-case'B':type_s="blob";break;-case'G':type_s="tag";break;-default:gotoerr_finish;-}-if(write_sha1_file(buffer,size,type_s,sha1)<0)+if(write_sha1_file(buffer,size,type,sha1)<0)die("failed to write %s (%s)",-sha1_to_hex(entry->sha1),type_s);-printf("%s %s\n",sha1_to_hex(sha1),type_s);+sha1_to_hex(entry->sha1),type);+printf("%s %s\n",sha1_to_hex(sha1),type);if(memcmp(sha1,entry->sha1,20))-die("resulting %s have wrong SHA1",type_s);+die("resulting %s have wrong SHA1",type);-finish:st=0;+finish:free(buffer);returnst;err_finish:
@@ -184,15 +185,13 @@ static int unpack_delta_entry(struct pacdie("truncated pack file");data=base_sha1+20;data_size=left-20;-printf("%s D %lu",sha1_to_hex(entry->sha1),delta_size);+printf("%s delta %lu",sha1_to_hex(entry->sha1),delta_size);printf(" %s\n",sha1_to_hex(base_sha1));if(dry_run)return0;-/* pack+5 is the base sha1, unless we have it, we need to-*unpackitfirst.-*/+/* unless we have the base sha1, we need to unpack it first. */if(!has_sha1_file(base_sha1)){structpack_entry*base;if(!find_pack_entry(base_sha1,&base))
@@ -237,7 +236,9 @@ static int unpack_delta_entry(struct pacstaticvoidunpack_entry(structpack_entry*entry){unsignedlongoffset,size,left;-unsignedchar*pack;+unsignedchar*pack,sizebits;+packed_obj_typetype;+inti;/* Have we done this one already due to deltas based on it? */if(lookup_object(entry->sha1))
@@ -247,17 +248,28 @@ static void unpack_entry(struct pack_entif(offset>pack_size-5)die("object offset outside of pack file");pack=pack_base+offset;-size=(pack[1]<<24)+(pack[2]<<16)+(pack[3]<<8)+pack[4];-left=pack_size-offset-5;-switch(*pack){-case'C':case'T':case'B':case'G':-unpack_non_delta_entry(entry,*pack,pack+5,size,left);+sizebits=*pack++;+type=sizebits&0x07;+size=(sizebits&~0x80)>>3;+i=4;+while(sizebits&0x80){+sizebits=*pack++;+size|=(sizebits&~0x80)<<i;+i+=7;+}+left=pack_size-((void*)pack-pack_base);+switch(type){+casePACKED_COMMIT:+casePACKED_TREE:+casePACKED_BLOB:+casePACKED_TAG:+unpack_non_delta_entry(entry,type_s[type],pack,size,left);break;-case'D':-unpack_delta_entry(entry,pack+5,size,left);+casePACKED_DELTA:+unpack_delta_entry(entry,pack,size,left);break;default:-die("corrupted pack file");+die("corrupted pack file(unknown object type %d)",type);}}
Dammit, I wasted all that time doing it myself.
I just committed and pushed out my version. But mine also does sha1_file.c
right, so that you can use a packed archive in .git/objects/pack. Yours
has some other cleanups, so..
Can you double-check my version (it hasn't mirrored out yet, it seems, but
it should be there soon).
Probably a version signature would be a good thing too.
I did that too, same format as for the index file. Nothing checks it
though.
Linus
From: Nicolas Pitre <hidden> Date: 2016-06-15 22:42:01
On Tue, 28 Jun 2005, Linus Torvalds wrote:
On Tue, 28 Jun 2005, Nicolas Pitre wrote:
quoted
OK. New patch below.
Dammit, I wasted all that time doing it myself.
I just committed and pushed out my version. But mine also does sha1_file.c
right, so that you can use a packed archive in .git/objects/pack. Yours
has some other cleanups, so..
Can you double-check my version (it hasn't mirrored out yet, it seems, but
it should be there soon).
OK... See below the cleanups I merged from my version on top of yours:
pack-objects.c | 70 ++++++++++++++-----------------------------------------
pack.h | 17 ++++++++-----
unpack-objects.c | 66 +++++++++++++++++++++++++--------------------------
3 files changed, 63 insertions(+), 90 deletions(-)
I also restored my original object header size ordering (little endian)
for two reasons:
- it is much simpler to generate and therefore allows for removing
quite some code
- it allows for stable bit position which makes it much easier to look
at an hex dump of the binary data for manual debugging
Also a few code optimizations and one error return fix.
Signed-off-by: Nicolas Pitre <redacted>
@@ -121,13 +128,6 @@ static int unpack_non_delta_entry(structinflateEnd(&stream);if((st!=Z_STREAM_END)||stream.total_out!=size)gotoerr_finish;-switch(kind){-caseOBJ_COMMIT:type="commit";break;-caseOBJ_TREE:type="tree";break;-caseOBJ_BLOB:type="blob";break;-caseOBJ_TAG:type="tag";break;-default:gotoerr_finish;-}if(write_sha1_file(buffer,size,type,sha1)<0)die("failed to write %s (%s)",sha1_to_hex(entry->sha1),type);
@@ -135,8 +135,8 @@ static int unpack_non_delta_entry(structif(memcmp(sha1,entry->sha1,20))die("resulting %s have wrong SHA1",type);-finish:st=0;+finish:free(buffer);returnst;err_finish:
@@ -185,15 +185,13 @@ static int unpack_delta_entry(struct pacdie("truncated pack file");data=base_sha1+20;data_size=left-20;-printf("%s D %lu",sha1_to_hex(entry->sha1),delta_size);+printf("%s delta %lu",sha1_to_hex(entry->sha1),delta_size);printf(" %s\n",sha1_to_hex(base_sha1));if(dry_run)return0;-/* pack+5 is the base sha1, unless we have it, we need to-*unpackitfirst.-*/+/* unless we have the base sha1, we need to unpack it first. */if(!has_sha1_file(base_sha1)){structpack_entry*base;if(!find_pack_entry(base_sha1,&base))
@@ -238,8 +236,9 @@ static int unpack_delta_entry(struct pacstaticvoidunpack_entry(structpack_entry*entry){unsignedlongoffset,size,left;-unsignedchar*pack,c;-inttype;+unsignedcharc,*pack=pack_base;+inti;+enumobject_typetype;/* Have we done this one already due to deltas based on it? */if(lookup_object(entry->sha1))
From: Nicolas Pitre <hidden> Date: 2016-06-15 22:42:01
On Tue, 28 Jun 2005, Nicolas Pitre wrote:
OK... See below the cleanups I merged from my version on top of yours:
Of course by the time I sent the above you already rewrote the ting to
be streamable.
So again :-) see below the cleanups I merged from my version on top of
yours:
pack-objects.c | 70 ++++++++++++++-----------------------------------------
pack.h | 17 ++++++++-----
unpack-objects.c | 29 ++++++++++++----------
3 files changed, 46 insertions(+), 70 deletions(-)
I also restored my original object header size ordering (little endian)
for two reasons:
- it is much simpler to generate and therefore allows for removing
quite some code
- it allows for stable bit position which makes it much easier to look
at an hex dump of the binary data for manual debugging
Signed-off-by: Nicolas Pitre <redacted>
Of course by the time I sent the above you already rewrote the ting to
be streamable.
And by the time you sent me a new version, I'd already taken part of your
old one by hand ;)
Btw, I have the size/type bits reversed from your setup, but please don't
change that, since that would be yet another incompatible pack format
change, and I'd like to calm things down.
Also, I notice that you decode the sizes really strangely: you have a
"while() { }" loop and two separate loads. It's much nicer to do it with a
"do { } while()" loop and a single load, since not only is it less code,
a do-while loop compiles to better code than a while() loop (unless the
compiler is crazy, which it sometimes is).
Linus