This contains many bug fixes or cleanups. Also you can now run the
test suite with v4 by setting GIT_TEST_OPTS=--packv4. The test suite
passes now. pack size limit is not officially not supported with v4.
index-pack also learns to convert appended trees to v4 for completing
thin packs (still need to convert commits though)
PS. Nico do you still take patches and then send pull requests to
Junio occasionally, or should I start to CC Junio?
Nguyễn Thái Ngọc Duy (17):
fixup! index-pack: record all delta bases in v4 (tree and ref-delta)
fixup! packv4-parse.c: add tree offset caching
fixup! pack-objects: support writing pack v4
fixup! pack-objects: recognize v4 as pack source
fixup! index-pack: support completing thin packs v4
fixup! pack v4: move packv4-create.c to libgit.a
fixup! index-pack, pack-objects: allow creating .idx v2 with .pack v4
fixup! pack v4: code to obtain a SHA1 from a sha1ref
fixup! pack-objects: add --version to specify written pack version
test-lib.sh: add --packv4 for running the tests with pack v4 as default
packv4-parse: accept ref-delta as base of pv4-tree
pack-objects: do not add type OBJ_NONE to objects[] in pack v4
index-pack: encode appended trees using v4 format in pack v4
t5302: disable sealth corruption tests when run with --packv4
t5300: avoid testing ofs-delta with --packv4
pack-objects: disable pack size limit feature on pack v4
t5303: adapt the tests to run with --packv4
.gitignore | 1 +
Documentation/git-pack-objects.txt | 4 +++
builtin/index-pack.c | 49 ++++++++++++++++++++++++++++++-----
builtin/pack-objects.c | 35 ++++++++++++++++++++-----
packv4-create.c | 20 ++++++++++++++
packv4-create.h | 2 ++
packv4-parse.c | 25 +++++++++++-------
sha1_file.c | 4 ++-
t/t5300-pack-object.sh | 47 ++++++++++++++++++---------------
t/t5302-pack-index.sh | 24 ++++++++---------
t/t5303-pack-corruption-resilience.sh | 16 ++++++------
t/test-lib.sh | 10 +++++++
12 files changed, 173 insertions(+), 64 deletions(-)
--
1.8.2.83.gc99314b
we need to pass an offset pointing to the header because
unpack_entry() needs that..
Signed-off-by: Nguyễn Thái Ngọc Duy <redacted>
---
packv4-parse.c | 5 ++---
1 file changed, 2 insertions(+), 3 deletions(-)
@@ -470,8 +470,7 @@ static int decode_entries(struct packed_git *p, struct pack_window **w_curs,/* is this a canonical tree object? */if((*scp&0xf)==OBJ_TREE){-offset=obj_offset+(scp-src);-returncopy_canonical_tree_entries(p,offset,+returncopy_canonical_tree_entries(p,obj_offset,start,count,dstp,sizep);}
@@ -775,8 +775,8 @@ static void *unpack_raw_entry(struct object_entry *obj,break;caseOBJ_OFS_DELTA:if(packv4)-die(_("pack version 4 does not support ofs-delta type (offset %lu)"),-(unsignedlong)obj->idx.offset);+bad_object(obj->idx.offset,+_("pack version 4 does not support ofs-delta type"));offset=obj->idx.offset-read_varint();if(offset<=0||offset>=obj->idx.offset)bad_object(obj->idx.offset,
In pack v4, commits should be forced to be loaded in canonical format
too (iow. deltified commits are flattened by read_sha1_file, we don't
care about object_entry->delta).
Signed-off-by: Nguyễn Thái Ngọc Duy <redacted>
---
builtin/pack-objects.c | 7 ++++---
1 file changed, 4 insertions(+), 3 deletions(-)
!is_delta_tree() is not correct because the tree may have been
resolved in by find_unresolved_deltas() in the previous iteration of
this loop. Check for entry->idx.sha1 instead, that must be non-null
when we resolve the object.
Signed-off-by: Nguyễn Thái Ngọc Duy <redacted>
---
builtin/index-pack.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
@@ -18,13 +18,18 @@ const unsigned char *get_sha1ref(struct packed_git *p,{constunsignedchar*sha1;+if(!p->sha1_table)+returnNULL;+if(!**bufp){sha1=*bufp+1;*bufp+=21;}else{unsignedintindex=decode_varint(bufp);-if(index<1||index-1>p->num_objects)-die("bad index in %s",__func__);+if(index<1||index-1>p->num_objects){+error("bad index in get_sha1ref");+returnNULL;+}sha1=p->sha1_table+(index-1)*20;}
--packv4 also sets prerequisite PACKV4, which can be used to disable
v2-specific tests. You can run the test suite with
make test GIT_TEST_OPTS=--packv4
or just a specific test
./t5300-*.sh --packv4 -v -i
Signed-off-by: Nguyễn Thái Ngọc Duy <redacted>
---
builtin/pack-objects.c | 4 ++++
t/test-lib.sh | 10 ++++++++++
2 files changed, 14 insertions(+)
@@ -222,6 +222,9 @@ do--statusprefix=*)statusprefix=$(expr"z$1":'z[^=]*=\(.*\)')shift;;+--packv4)+packv4=t+shift;;*)echo"error: unknown test option '$1'">&2;exit1;;esac
@@ -740,6 +743,13 @@ elsemkdir-p"$TRASH_DIRECTORY"fi+iftest-n"$packv4"+then+GIT_TEST_PACKV4=t+exportGIT_TEST_PACKV4+test_set_prereqPACKV4+fi+# Gross hack to spawn N sub-instances of the tests in parallel, and# summarize the results. Note that if this is enabled, the script# terminates at the end of this 'if' block.
@@ -220,6 +220,10 @@ So does `git bundle` (see linkgit:git-bundle[1]) when it creates a bundle. to force the version for the generated pack index, and to force 64-bit index entries on objects located above the given offset.+--version=<version>::+ Force the version for the generated pack, Valid values are 2,+ 3 and 4. Default is 2.+ --keep-true-parents:: With this option, parents that are hidden by grafts are packed nevertheless.
@@ -473,16 +473,19 @@ static int decode_entries(struct packed_git *p, struct pack_window **w_curs,if(++scp-src>=avail-20)return-1;+switch(*scp++&0xf){/* is this a canonical tree object? */-if((*scp&0xf)==OBJ_TREE){+caseOBJ_TREE:+caseOBJ_REF_DELTA:returncopy_canonical_tree_entries(p,obj_offset,start,count,dstp,sizep);-}-/* let's still make sure this is actually a pv4 tree */-if((*scp++&0xf)!=OBJ_PV4_TREE)+caseOBJ_PV4_TREE:+break;+default:return-1;+}nb_entries=decode_varint(&scp);if(!count)
This is a longer explation of what is noted in the patch. When object
names are received from stdin, we lazily put OBJ_NONE as type to
objects[]. check_object() is called for each entry in objects[] later,
when it checks for ref-delta and ofs-delta for straight copy to the
pack later.
In pack v4, we don't store commits and trees as ref-delta and we need
a way to know those are delta objects in pack v2 sources are actually
commits or trees. We detect that with "type" field in struct
object_entry, which is correctly filled when --revs is passed. Without
--revs, the "type" field would be OBJ_NONE and we would need another
sha1_object_info() or similar call to detect the true object type.
Because we need object type anyway in this code path for building up
ident and path dictionaries, fill correct type as well.
Without this, the condition "pack_version < 4 || entry->type !=
OBJ_TREE" in check_object() is always true when --revs is not used,
and we will encode trees as OBJ_REF_DELTA with the base either
OBJ_TREE or OBJ_PV4_TREE. That kills pv4 tree format advantages.
Signed-off-by: Nguyễn Thái Ngọc Duy <redacted>
---
builtin/pack-objects.c | 17 ++++++++++++++---
1 file changed, 14 insertions(+), 3 deletions(-)
@@ -1717,18 +1719,48 @@ static int write_compressed(struct sha1file *f, void *in, unsigned int size)returnsize;}+staticvoidinitialize_packv4_tables(void)+{+staticintinitialized;+inti,nr=nr_objects_final;+if(initialized)+return;+pv4_tables.commit_ident_table=pv4_dict_to_dict_table(name_dict);+pv4_tables.tree_path_table=pv4_dict_to_dict_table(path_dict);+pv4_tables.all_objs_nr=nr;+pv4_tables.all_objs=xmalloc(nr*sizeof(structpack_idx_entry));+/* for pv4_encode_tree() pv4_tables[].offset is not needed */+for(i=0;i<nr;i++)+hashcpy(pv4_tables.all_objs[i].sha1,sha1_table+i*20);+initialized=1;+}+staticstructobject_entry*append_obj_to_pack(structsha1file*f,constunsignedchar*sha1,void*buf,unsignedlongsize,enumobject_typetype){structobject_entry*obj=&objects[nr_objects++];+void*v4_data=NULL;unsignedcharheader[10];+unsignedlongv4_size;+enumobject_typereal_type=type;intn;if(packv4){if(nr_objects>nr_objects_final)die(_("too many objects"));-/* TODO: convert OBJ_TREE to OBJ_PV4_TREE using pv4_encode_tree */++if(type==OBJ_TREE){+initialize_packv4_tables();+v4_size=size;+v4_data=pv4_encode_tree(&pv4_tables,buf,&v4_size,+NULL,0,NULL);+if(v4_data)+type=OBJ_PV4_TREE;+}++/* TODO: convert OBJ_COMMIT to OBJ_PV4_COMMIT using pv4_encode_commit */+n=pv4_encode_object_header(type,size,header);}elsen=encode_in_pack_object_header(type,size,header);
@@ -1737,12 +1769,17 @@ static struct object_entry *append_obj_to_pack(struct sha1file *f,obj[0].size=size;obj[0].hdr_size=n;obj[0].type=type;-obj[0].real_type=type;+obj[0].real_type=real_type;obj[1].idx.offset=obj[0].idx.offset+n;-obj[1].idx.offset+=write_compressed(f,buf,size);+if(type!=real_type){/* must be v4 representation */+sha1write(f,v4_data,v4_size);+obj[1].idx.offset+=v4_size;+}else+obj[1].idx.offset+=write_compressed(f,buf,size);obj[0].idx.crc32=crc32_end(f);sha1flush(f);hashcpy(obj->idx.sha1,sha1);+free(v4_data);returnobj;}
@@ -144,6 +145,25 @@ void sort_dict_entries_by_hits(struct dict_table *t)rehash_entries(t);}+structdict_table*pv4_dict_to_dict_table(structpackv4_dict*pv4dict)+{+structdict_table*dict;+inti;++dict=create_dict_table();+for(i=0;i<pv4dict->nb_entries;i++){+constunsignedchar*mode_bytes;+constchar*str;+intmode,str_len;+mode_bytes=pv4dict->data+pv4dict->offsets[i];+mode=(mode_bytes[0]<<8)|mode_bytes[1];+str=(constchar*)mode_bytes+2;+str_len=pv4dict->offsets[i+1]-pv4dict->offsets[i]-2;+str_len--;/* for NUL, dict_add_entry will add one back */+dict_add_entry(dict,mode,str,str_len);+}+returndict;+}/**Parsetheauthor/committerlinefromacanonicalcommitobject.*The'from'argumentpointsrightafterthe"author "or"committer "
These tests assume .pack v2 format and won't work with v4. New tests
may be needed to do the same thing with v4 format.
Signed-off-by: Nguyễn Thái Ngọc Duy <redacted>
---
t/t5302-pack-index.sh | 24 ++++++++++++------------
1 file changed, 12 insertions(+), 12 deletions(-)
@@ -148,7 +148,7 @@ test_expect_success \cmp"test-1-${pack1}.pack"".git/objects/pack/pack-${pack1}.pack"&&cmp"test-1-${pack1}.idx"".git/objects/pack/pack-${pack1}.idx"'-test_expect_success\+test_expect_success!PACKV4\'[index v1] 2) create a stealth corruption in a delta base reference'\'# This test assumes file_101 is a delta smaller than 16 bytes.# It should be against file_100 but we substitute its base for file_099
@@ -163,24 +163,24 @@ test_expect_success \bs=1count=20conv=notrunc&&gitcat-fileblob$sha1_101>file_101_foo1'-test_expect_success\+test_expect_success!PACKV4\'[index v1] 3) corrupted delta happily returned wrong data'\'test -f file_101_foo1 && ! cmp file_101 file_101_foo1'-test_expect_success\+test_expect_success!PACKV4\'[index v1] 4) confirm that the pack is actually corrupted'\'test_must_fail git fsck --full $commit'-test_expect_success\+test_expect_success!PACKV4\'[index v1] 5) pack-objects happily reuses corrupted data'\'pack4=$(gitpack-objectstest-4<obj-list)&&test-f"test-4-${pack1}.pack"'-test_expect_success\+test_expect_success!PACKV4\'[index v1] 6) newly created pack is BAD !'\'test_must_fail git verify-pack -v "test-4-${pack1}.pack"'-test_expect_success\+test_expect_success!PACKV4\'[index v2] 1) stream pack to repository'\'rm-f.git/objects/pack/*&&gitindex-pack--index-version=2--stdin<"test-1-${pack1}.pack"&&
@@ -189,7 +189,7 @@ test_expect_success \cmp"test-1-${pack1}.pack"".git/objects/pack/pack-${pack1}.pack"&&cmp"test-2-${pack1}.idx"".git/objects/pack/pack-${pack1}.idx"'-test_expect_success\+test_expect_success!PACKV4\'[index v2] 2) create a stealth corruption in a delta base reference'\'# This test assumes file_101 is a delta smaller than 16 bytes.# It should be against file_100 but we substitute its base for file_099
@@ -204,20 +204,20 @@ test_expect_success \bs=1count=20conv=notrunc&&gitcat-fileblob$sha1_101>file_101_foo2'-test_expect_success\+test_expect_success!PACKV4\'[index v2] 3) corrupted delta happily returned wrong data'\'test -f file_101_foo2 && ! cmp file_101 file_101_foo2'-test_expect_success\+test_expect_success!PACKV4\'[index v2] 4) confirm that the pack is actually corrupted'\'test_must_fail git fsck --full $commit'-test_expect_success\+test_expect_success!PACKV4\'[index v2] 5) pack-objects refuses to reuse corrupted data'\'test_must_failgitpack-objectstest-5<obj-list&&test_must_failgitpack-objects--no-reuse-objecttest-6<obj-list'-test_expect_success\+test_expect_success!PACKV4\'[index v2] 6) verify-pack detects CRC mismatch'\'rm-f.git/objects/pack/*&&gitindex-pack--index-version=2--stdin<"test-1-${pack1}.pack"&&
pack v4 format does not go along well with pack size limit feature. v4
requires to know the number of objects in the pack in advance in order
to contruct various tables after pack header. When pack size is
limited, we don't know this number until we write objects out and hit
it. By then the only option we have is rewrite the pack and update the
tables.
Signed-off-by: Nguyễn Thái Ngọc Duy <redacted>
---
builtin/pack-objects.c | 2 ++
t/t5300-pack-object.sh | 8 ++++----
2 files changed, 6 insertions(+), 4 deletions(-)
@@ -2804,6 +2804,8 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)warning("minimum pack size limit is 1 MiB");pack_size_limit=1024*1024;}+if(pack_size_limit&&pack_version>=4)+die("pack size limiting is not supported with pack version 4");if(!pack_to_stdout&&thin)die("--thin cannot be used to build an indexable pack.");
"git show-index" does not work with .idx v3 which is the default for
.pack v4. Force .idx back to v2. Disable tests about OBJ_OFS_DELTA on
v4 because that is not supported.
Signed-off-by: Nguyễn Thái Ngọc Duy <redacted>
---
t/t5303-pack-corruption-resilience.sh | 16 ++++++++--------
1 file changed, 8 insertions(+), 8 deletions(-)
@@ -205,7 +205,7 @@ test_expect_success \gitcat-fileblob$blob_2>/dev/null&&gitcat-fileblob$blob_3>/dev/null'-test_expect_success\+test_expect_success!PACKV4\'corruption #0 in delta base reference of first delta (OBJ_OFS_DELTA)'\'create_new_pack--delta-base-offset&&gitprune-packed&&
@@ -214,7 +214,7 @@ test_expect_success \test_must_failgitcat-fileblob$blob_2>/dev/null&&test_must_failgitcat-fileblob$blob_3>/dev/null'-test_expect_success\+test_expect_success!PACKV4\'... but having a loose copy allows for full recovery'\'mv${pack}.idxtmp&&githash-object-tblob-wfile_2&&
@@ -223,7 +223,7 @@ test_expect_success \gitcat-fileblob$blob_2>/dev/null&&gitcat-fileblob$blob_3>/dev/null'-test_expect_success\+test_expect_success!PACKV4\'... and then a repack "clears" the corruption'\'do_repack--delta-base-offset&&gitprune-packed&&
@@ -232,7 +232,7 @@ test_expect_success \gitcat-fileblob$blob_2>/dev/null&&gitcat-fileblob$blob_3>/dev/null'-test_expect_success\+test_expect_success!PACKV4\'corruption #1 in delta base reference of first delta (OBJ_OFS_DELTA)'\'create_new_pack--delta-base-offset&&gitprune-packed&&
@@ -241,7 +241,7 @@ test_expect_success \test_must_failgitcat-fileblob$blob_2>/dev/null&&test_must_failgitcat-fileblob$blob_3>/dev/null'-test_expect_success\+test_expect_success!PACKV4\'... but having a loose copy allows for full recovery'\'mv${pack}.idxtmp&&githash-object-tblob-wfile_2&&
@@ -250,7 +250,7 @@ test_expect_success \gitcat-fileblob$blob_2>/dev/null&&gitcat-fileblob$blob_3>/dev/null'-test_expect_success\+test_expect_success!PACKV4\'... and then a repack "clears" the corruption'\'do_repack--delta-base-offset&&gitprune-packed&&
@@ -259,7 +259,7 @@ test_expect_success \gitcat-fileblob$blob_2>/dev/null&&gitcat-fileblob$blob_3>/dev/null'-test_expect_success\+test_expect_success!PACKV4\'... and a redundant pack allows for full recovery too'\'do_corrupt_object$blob_22<zero&&gitcat-fileblob$blob_1>/dev/null&&
From: Nicolas Pitre <nico@fluxnic.net> Date: 2016-06-15 22:58:52
On Sat, 21 Sep 2013, Nguyễn Thái Ngọc Duy wrote:
This contains many bug fixes or cleanups. Also you can now run the
test suite with v4 by setting GIT_TEST_OPTS=--packv4. The test suite
passes now. pack size limit is not officially not supported with v4.
index-pack also learns to convert appended trees to v4 for completing
thin packs (still need to convert commits though)
PS. Nico do you still take patches and then send pull requests to
Junio occasionally, or should I start to CC Junio?
I'm still willing to act as the middle man if that suits everybody.
That gives me the opportunity to review those patches and stay minimally
involved.
Nicolas
[...]
This, in combination with patch 10/17, is making the test suite to test
either packv4 or non packv4, and never both. I think this is not a good
approach.
Instead we should have packv2 specific tests to enforce --pack-version=2
when using pack-objects and create a duplicate of those tests for
--pack-version=4 when that makes sense. For tests that are mostly
common, the test could be factored out into a function with a pack
version argument. Then, most tests could be always run twice: once for
packv2 and again for packv4. Not doing so makes it more risky to
regress packv2 when testing improvements to packv4 support.
Nicolas
I folded it into "pack v4: initial pack index v3 support on the read
side" rather than "index-pack, pack-objects: allow creating .idx v2 with
.pack v4".
Nicolas
[...]
This, in combination with patch 10/17, is making the test suite to test
either packv4 or non packv4, and never both. I think this is not a good
approach.
Instead we should have packv2 specific tests to enforce --pack-version=2
when using pack-objects and create a duplicate of those tests for
--pack-version=4 when that makes sense. For tests that are mostly
common, the test could be factored out into a function with a pack
version argument. Then, most tests could be always run twice: once for
packv2 and again for packv4. Not doing so makes it more risky to
regress packv2 when testing improvements to packv4 support.
I agree. I wanted to split this (and maybe other t53xx) for v4-only
tests and update the existing t53xx to test on v2 only. For now I
think this will do as it will allow us to verify that v4 code works
(just need to run the test suite twice, with and without --packv4).
I'll add more v4 tests that run without --packv4. 10/17 should remain
in the end though to exercise v4 a lot more (some v4 bugs were found
not by t53xx), until we finally switch the default to v4.
--
Duy
[...]
This, in combination with patch 10/17, is making the test suite to test
either packv4 or non packv4, and never both. I think this is not a good
approach.
Instead we should have packv2 specific tests to enforce --pack-version=2
when using pack-objects and create a duplicate of those tests for
--pack-version=4 when that makes sense. For tests that are mostly
common, the test could be factored out into a function with a pack
version argument. Then, most tests could be always run twice: once for
packv2 and again for packv4. Not doing so makes it more risky to
regress packv2 when testing improvements to packv4 support.
I agree. I wanted to split this (and maybe other t53xx) for v4-only
tests and update the existing t53xx to test on v2 only. For now I
think this will do as it will allow us to verify that v4 code works
(just need to run the test suite twice, with and without --packv4).
OK.
I've queued those patches at the end of the series so they're easily
replaceable.
Nicolas
From: Eric Sunshine <hidden> Date: 2016-06-15 22:58:52
On Sat, Sep 21, 2013 at 9:58 AM, Nguyễn Thái Ngọc Duy [off-list ref] wrote:
[PATCH 14/17] t5302: disable sealth corruption tests when run with --packv4
s/sealth/stealth/
These tests assume .pack v2 format and won't work with v4. New tests
may be needed to do the same thing with v4 format.
Signed-off-by: Nguyễn Thái Ngọc Duy <redacted>