From: Stefan Beller <hidden> Date: 2018-12-15 00:09:48
I realized next has not been rewound, so I can resend sb/more-repo-in-api,
which I hereby do. The changes are minimal and address the only comment
by Jonathan so far.
Thanks,
Stefan
Stefan Beller (23):
sha1_file: allow read_object to read objects in arbitrary repositories
packfile: allow has_packed_and_bad to handle arbitrary repositories
object-store: allow read_object_file_extended to read from any repo
object-store: prepare read_object_file to deal with any repo
object-store: prepare has_{sha1, object}_file to handle any repo
object: parse_object to honor its repository argument
commit: allow parse_commit* to handle any repo
commit-reach.c: allow paint_down_to_common to handle any repo
commit-reach.c: allow merge_bases_many to handle any repo
commit-reach.c: allow remove_redundant to handle any repo
commit-reach.c: allow get_merge_bases_many_0 to handle any repo
commit-reach: prepare get_merge_bases to handle any repo
commit-reach: prepare in_merge_bases[_many] to handle any repo
commit: prepare get_commit_buffer to handle any repo
commit: prepare repo_unuse_commit_buffer to handle any repo
commit: prepare logmsg_reencode to handle arbitrary repositories
pretty: prepare format_commit_message to handle arbitrary repositories
submodule: use submodule repos for object lookup
submodule: don't add submodule as odb for push
commit-graph: convert remaining functions to handle any repo
commit: prepare free_commit_buffer and release_commit_memory for any
repo
path.h: make REPO_GIT_PATH_FUNC repository agnostic
t/helper/test-repository: celebrate independence from the_repository
builtin/fsck.c | 3 +-
builtin/log.c | 6 +-
builtin/rev-list.c | 3 +-
cache.h | 2 +
commit-graph.c | 40 +++--
commit-reach.c | 73 +++++----
commit-reach.h | 38 +++--
commit.c | 41 ++---
commit.h | 43 +++++-
.../coccinelle/the_repository.pending.cocci | 144 ++++++++++++++++++
object-store.h | 35 ++++-
object.c | 8 +-
packfile.c | 5 +-
packfile.h | 2 +-
path.h | 2 +-
pretty.c | 28 ++--
pretty.h | 7 +-
sha1-file.c | 34 +++--
streaming.c | 2 +-
submodule.c | 78 +++++++---
t/helper/test-repository.c | 10 ++
21 files changed, 454 insertions(+), 150 deletions(-)
create mode 100644 contrib/coccinelle/the_repository.pending.cocci
git range-diff origin/sb/more-repo-in-api... >>0000-cover-letter.patch
1: 99017ffac8 ! 1: f24b120287 submodule: use submodule repos for object lookup
@@ -40,12 +40,13 @@
- * attempt to lookup both the left and right commits and put them into the
- * left and right pointers.
+/*
-+ * Initialize 'out' based on the provided submodule path.
++ * Initialize a repository struct for a submodule based on the provided 'path'.
+ *
+ * Unlike repo_submodule_init, this tolerates submodules not present
+ * in .gitmodules. This function exists only to preserve historical behavior,
+ *
-+ * Returns 0 on success, -1 when the submodule is not present.
++ * Returns the repository struct on success,
++ * NULL when the submodule is not present.
*/
-static void show_submodule_header(struct diff_options *o, const char *path,
+static struct repository *open_submodule(const char *path)
@@ -59,6 +60,7 @@
+ return NULL;
+ }
+
++ /* Mark it as a submodule */
+ out->submodule_prefix = xstrdup(path);
+
+ strbuf_release(&sb);
2: 809765861c = 2: 25190d6174 submodule: don't add submodule as odb for push
3: 4a7735da72 = 3: 965421aab2 commit-graph: convert remaining functions to handle any repo
4: aeeb1ba49e = 4: bf31f32723 commit: prepare free_commit_buffer and release_commit_memory for any repo
5: 5ffebe9463 = 5: c4e54e6b0d path.h: make REPO_GIT_PATH_FUNC repository agnostic
6: 9c89920c46 = 6: a7ed0c57ba t/helper/test-repository: celebrate independence from the_repository
From: Stefan Beller <hidden> Date: 2018-12-15 00:09:50
Allow read_object (a file local functon in sha1_file) to
handle arbitrary repositories by passing the repository down
to oid_object_info_extended.
Signed-off-by: Stefan Beller <redacted>
Signed-off-by: Junio C Hamano <redacted>
---
sha1-file.c | 10 ++++++----
1 file changed, 6 insertions(+), 4 deletions(-)
From: Stefan Beller <hidden> Date: 2018-12-15 00:09:52
has_packed_and_bad is not widely used, so just migrate it all at once.
Signed-off-by: Stefan Beller <redacted>
Signed-off-by: Junio C Hamano <redacted>
---
packfile.c | 5 +++--
packfile.h | 2 +-
sha1-file.c | 2 +-
3 files changed, 5 insertions(+), 4 deletions(-)
From: Stefan Beller <hidden> Date: 2018-12-15 00:09:54
read_object_file_extended is not widely used, so migrate it all at once.
Signed-off-by: Stefan Beller <redacted>
Signed-off-by: Junio C Hamano <redacted>
---
object-store.h | 5 +++--
sha1-file.c | 11 ++++++-----
streaming.c | 2 +-
3 files changed, 10 insertions(+), 8 deletions(-)
@@ -1428,11 +1429,11 @@ void *read_object_file_extended(const struct object_id *oid,die(_("replacement %s not found for %s"),oid_to_hex(repl),oid_to_hex(oid));-if(!stat_sha1_file(the_repository,repl->hash,&st,&path))+if(!stat_sha1_file(r,repl->hash,&st,&path))die(_("loose object %s (stored in %s) is corrupt"),oid_to_hex(repl),path);-if((p=has_packed_and_bad(the_repository,repl->hash))!=NULL)+if((p=has_packed_and_bad(r,repl->hash))!=NULL)die(_("packed object %s (stored in %s) is corrupt"),oid_to_hex(repl),p->pack_name);
From: Stefan Beller <hidden> Date: 2018-12-15 00:09:57
As read_object_file is a widely used function (which is also regularly used
in new code in flight between master..pu), changing its signature is painful
is hard, as other series in flight rely on the original signature. It would
burden the maintainer if we'd just change the signature.
Introduce repo_read_object_file which takes the repository argument, and
hide the original read_object_file as a macro behind
NO_THE_REPOSITORY_COMPATIBILITY_MACROS, similar to
e675765235 (diff.c: remove implicit dependency on the_index, 2018-09-21)
Add a coccinelle patch to convert existing callers, but do not apply
the resulting patch to keep the diff of this patch small.
Signed-off-by: Stefan Beller <redacted>
Signed-off-by: Junio C Hamano <redacted>
---
contrib/coccinelle/the_repository.pending.cocci | 12 ++++++++++++
object-store.h | 10 ++++++++--
2 files changed, 20 insertions(+), 2 deletions(-)
create mode 100644 contrib/coccinelle/the_repository.pending.cocci
@@ -0,0 +1,12 @@+// This file is used for the ongoing refactoring of+// bringing the index or repository struct in all of+// our code base.++@@+expression E;+expression F;+expression G;+@@+- read_object_file(++ repo_read_object_file(the_repository,+ E, F, G)
From: Stefan Beller <hidden> Date: 2018-12-15 00:10:01
In 8e4b0b6047 (object.c: allow parse_object to handle
arbitrary repositories, 2018-06-28), we forgot to pass the
repository down to the read_object_file.
Signed-off-by: Stefan Beller <redacted>
Signed-off-by: Junio C Hamano <redacted>
---
object.c | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
From: Stefan Beller <hidden> Date: 2018-12-15 00:10:04
Just like the previous commit, parse_commit and friends are used a lot
and are found in new patches, so we cannot change their signature easily.
Re-introduce these function prefixed with 'repo_' that take a repository
argument and keep the original as a shallow macro.
Signed-off-by: Stefan Beller <redacted>
Signed-off-by: Junio C Hamano <redacted>
---
commit.c | 18 ++++++++------
commit.h | 17 +++++++++----
.../coccinelle/the_repository.pending.cocci | 24 +++++++++++++++++++
3 files changed, 48 insertions(+), 11 deletions(-)
@@ -454,9 +457,9 @@ int parse_commit_internal(struct commit *item, int quiet_on_missing, int use_comreturn-1;if(item->object.parsed)return0;-if(use_commit_graph&&parse_commit_in_graph(the_repository,item))+if(use_commit_graph&&parse_commit_in_graph(r,item))return0;-buffer=read_object_file(&item->object.oid,&type,&size);+buffer=repo_read_object_file(r,&item->object.oid,&type,&size);if(!buffer)returnquiet_on_missing?-1:error("Could not read %s",
@@ -467,18 +470,19 @@ int parse_commit_internal(struct commit *item, int quiet_on_missing, int use_comoid_to_hex(&item->object.oid));}-ret=parse_commit_buffer(the_repository,item,buffer,size,0);+ret=parse_commit_buffer(r,item,buffer,size,0);if(save_commit_buffer&&!ret){-set_commit_buffer(the_repository,item,buffer,size);+set_commit_buffer(r,item,buffer,size);return0;}free(buffer);returnret;}-intparse_commit_gently(structcommit*item,intquiet_on_missing)+intrepo_parse_commit_gently(structrepository*r,+structcommit*item,intquiet_on_missing){-returnparse_commit_internal(item,quiet_on_missing,1);+returnrepo_parse_commit_internal(r,item,quiet_on_missing,1);}voidparse_commit_or_die(structcommit*item)
From: Stefan Beller <hidden> Date: 2018-12-15 00:10:07
As the function is file local and not widely used, migrate it all at once.
Signed-off-by: Stefan Beller <redacted>
Signed-off-by: Junio C Hamano <redacted>
---
commit-reach.c | 15 +++++++++------
1 file changed, 9 insertions(+), 6 deletions(-)
@@ -30,7 +30,8 @@ static int queue_has_nonstale(struct prio_queue *queue)}/* all input commits in one and twos[] must have been parsed! */-staticstructcommit_list*paint_down_to_common(structcommit*one,intn,+staticstructcommit_list*paint_down_to_common(structrepository*r,+structcommit*one,intn,structcommit**twos,intmin_generation){
From: Stefan Beller <hidden> Date: 2018-12-15 00:10:16
Similarly to previous patches, the get_merge_base functions are used
often in the code base, which makes migrating them hard.
Implement the new functions, prefixed with 'repo_' and hide the old
functions behind a wrapper macro.
Signed-off-by: Stefan Beller <redacted>
Signed-off-by: Junio C Hamano <redacted>
---
commit-reach.c | 24 ++++++++++-------
commit-reach.h | 26 ++++++++++++-------
.../coccinelle/the_repository.pending.cocci | 26 +++++++++++++++++++
3 files changed, 56 insertions(+), 20 deletions(-)
@@ -8,17 +8,23 @@ struct commit_list;structcontains_cache;structref_filter;-structcommit_list*get_merge_bases_many(structcommit*one,-intn,-structcommit**twos);-structcommit_list*get_merge_bases_many_dirty(structcommit*one,-intn,-structcommit**twos);-structcommit_list*get_merge_bases(structcommit*one,structcommit*two);-structcommit_list*get_octopus_merge_bases(structcommit_list*in);-+structcommit_list*repo_get_merge_bases(structrepository*r,+structcommit*rev1,+structcommit*rev2);+structcommit_list*repo_get_merge_bases_many(structrepository*r,+structcommit*one,intn,+structcommit**twos);/* To be used only when object flags after this call no longer matter */-structcommit_list*get_merge_bases_many_dirty(structcommit*one,intn,structcommit**twos);+structcommit_list*repo_get_merge_bases_many_dirty(structrepository*r,+structcommit*one,intn,+structcommit**twos);+#ifndef NO_THE_REPOSITORY_COMPATIBILITY_MACROS+#define get_merge_bases(r1, r2) repo_get_merge_bases(the_repository, r1, r2)+#define get_merge_bases_many(one, n, two) repo_get_merge_bases_many(the_repository, one, n, two)+#define get_merge_bases_many_dirty(one, n, twos) repo_get_merge_bases_many_dirty(the_repository, one, n, twos)+#endif++structcommit_list*get_octopus_merge_bases(structcommit_list*in);intis_descendant_of(structcommit*commit,structcommit_list*with_commit);intin_merge_bases_many(structcommit*commit,intnr_reference,structcommit**reference);
@@ -180,6 +180,14 @@ extern int has_non_ascii(const char *text);externconstchar*logmsg_reencode(conststructcommit*commit,char**commit_encoding,constchar*output_encoding);+constchar*repo_logmsg_reencode(structrepository*r,+conststructcommit*commit,+char**commit_encoding,+constchar*output_encoding);+#ifndef NO_THE_REPOSITORY_COMPATIBILITY_MACROS+#define logmsg_reencode(c, enc, out) repo_logmsg_reencode(the_repository, c, enc, out)+#endif+externconstchar*skip_blank_lines(constchar*msg);/** Removes the first commit from a list sorted by date, and adds all
From: Stefan Beller <hidden> Date: 2018-12-15 00:10:30
This converts the 'show_submodule_header' function to use
the repository API properly, such that the submodule objects
are not added to the main object store.
Signed-off-by: Stefan Beller <redacted>
Signed-off-by: Junio C Hamano <redacted>
---
submodule.c | 75 ++++++++++++++++++++++++++++++++++++++++++-----------
1 file changed, 60 insertions(+), 15 deletions(-)
@@ -481,14 +482,46 @@ void prepare_submodule_repo_env(struct argv_array *out)DEFAULT_GIT_DIR_ENVIRONMENT);}-/* Helper function to display the submodule header line prior to the full-*summaryoutput.Ifitcanlocatethesubmoduleobjectsdirectoryitwill-*attempttolookupboththeleftandrightcommitsandputthemintothe-*leftandrightpointers.+/*+*Initializearepositorystructforasubmodulebasedontheprovided'path'.+*+*Unlikerepo_submodule_init,thistoleratessubmodulesnotpresent+*in.gitmodules.Thisfunctionexistsonlytopreservehistoricalbehavior,+*+*Returnstherepositorystructonsuccess,+*NULLwhenthesubmoduleisnotpresent.*/-staticvoidshow_submodule_header(structdiff_options*o,constchar*path,+staticstructrepository*open_submodule(constchar*path)+{+structstrbufsb=STRBUF_INIT;+structrepository*out=xmalloc(sizeof(*out));++if(submodule_to_gitdir(&sb,path)||repo_init(out,sb.buf,NULL)){+strbuf_release(&sb);+free(out);+returnNULL;+}++/* Mark it as a submodule */+out->submodule_prefix=xstrdup(path);++strbuf_release(&sb);+returnout;+}++/*+*Helperfunctiontodisplaythesubmoduleheaderlinepriortothefull+*summaryoutput.+*+*Ifitcanlocatethesubmodulegitdirectoryitwillcreatearepository+*handleforthesubmoduleandlookupboththeleftandrightcommitsand+*putthemintotheleftandrightpointers.+*/+staticvoidshow_submodule_header(structdiff_options*o,+constchar*path,structobject_id*one,structobject_id*two,unsigneddirty_submodule,+structrepository*sub,structcommit**left,structcommit**right,structcommit_list**merge_bases){
@@ -598,9 +637,11 @@ void show_submodule_inline_diff(struct diff_options *o, const char *path,structcommit_list*merge_bases=NULL;structchild_processcp=CHILD_PROCESS_INIT;structstrbufsb=STRBUF_INIT;+structrepository*sub;+sub=open_submodule(path);show_submodule_header(o,path,one,two,dirty_submodule,-&left,&right,&merge_bases);+sub,&left,&right,&merge_bases);/* We need a valid left and right commit to display a difference */if(!(left||is_null_oid(one))||
From: Stefan Beller <hidden> Date: 2018-12-15 00:10:32
In push_submodule(), because we do not actually need access to objects
in the submodule, do not invoke add_submodule_odb().
(for_each_remote_ref_submodule() does not require access to those
objects, and the actual push is done by spawning another process,
which handles object access by itself.)
This code of push_submodule() is exercised in t5531 and continues
to work, showing that the submodule odbc is not needed.
Signed-off-by: Stefan Beller <redacted>
Signed-off-by: Junio C Hamano <redacted>
---
submodule.c | 3 ---
1 file changed, 3 deletions(-)
From: Stefan Beller <hidden> Date: 2018-12-15 00:10:35
Convert all functions to handle arbitrary repositories in commit-graph.c
that are used by functions taking a repository argument already.
Notable exclusion is write_commit_graph and its local functions as that
only works on the_repository.
Signed-off-by: Stefan Beller <redacted>
Signed-off-by: Junio C Hamano <redacted>
---
commit-graph.c | 40 ++++++++++++++++++++++++----------------
1 file changed, 24 insertions(+), 16 deletions(-)
@@ -423,12 +431,12 @@ static struct tree *get_commit_tree_in_graph_one(struct commit_graph *g,if(c->graph_pos==COMMIT_NOT_FROM_GRAPH)BUG("get_commit_tree_in_graph_one called from non-commit-graph commit");-returnload_tree_for_commit(g,(structcommit*)c);+returnload_tree_for_commit(r,g,(structcommit*)c);}structtree*get_commit_tree_in_graph(structrepository*r,conststructcommit*c){-returnget_commit_tree_in_graph_one(r->objects->commit_graph,c);+returnget_commit_tree_in_graph_one(r,r->objects->commit_graph,c);}staticvoidwrite_graph_chunk_fanout(structhashfile*f,
@@ -1025,7 +1033,7 @@ int verify_commit_graph(struct repository *r, struct commit_graph *g)}graph_commit=lookup_commit(r,&cur_oid);-if(!parse_commit_in_graph_one(g,graph_commit))+if(!parse_commit_in_graph_one(r,g,graph_commit))graph_report("failed to parse %s from commit-graph",oid_to_hex(&cur_oid));}
@@ -1061,7 +1069,7 @@ int verify_commit_graph(struct repository *r, struct commit_graph *g)continue;}-if(!oideq(&get_commit_tree_in_graph_one(g,graph_commit)->object.oid,+if(!oideq(&get_commit_tree_in_graph_one(r,g,graph_commit)->object.oid,get_commit_tree_oid(odb_commit)))graph_report("root tree OID for commit %s in commit-graph is %s != %s",oid_to_hex(&cur_oid),
From: Stefan Beller <hidden> Date: 2018-12-15 00:10:38
Pass the object pool to free_commit_buffer and release_commit_memory,
such that we can eliminate access to 'the_repository'.
Also remove the TODO in release_commit_memory, as commit->util was
removed in 9d2c97016f (commit.h: delete 'util' field in struct commit,
2018-05-19)
Signed-off-by: Stefan Beller <redacted>
Signed-off-by: Junio C Hamano <redacted>
---
builtin/fsck.c | 3 ++-
builtin/log.c | 6 ++++--
builtin/rev-list.c | 3 ++-
commit.c | 9 ++++-----
commit.h | 4 ++--
object.c | 2 +-
6 files changed, 15 insertions(+), 12 deletions(-)
@@ -395,7 +395,8 @@ static int cmd_log_walk(struct rev_info *rev)*Wemayshowagivencommitmultipletimeswhen*walkingthereflogs.*/-free_commit_buffer(commit);+free_commit_buffer(the_repository->parsed_objects,+commit);free_commit_list(commit->parents);commit->parents=NULL;}
@@ -1922,7 +1923,8 @@ int cmd_format_patch(int argc, const char **argv, const char *prefix)open_next_file(rev.numbered_files?NULL:commit,NULL,&rev,quiet))die(_("Failed to create output files"));shown=log_tree_commit(&rev,commit);-free_commit_buffer(commit);+free_commit_buffer(the_repository->parsed_objects,+commit);/* We put one extra blank line between formatted*patchesandthisflagisusedbylog-treecode
From: Stefan Beller <hidden> Date: 2018-12-15 00:10:40
git_pathdup uses the_repository internally, but the macro
REPO_GIT_PATH_FUNC is specifically made for arbitrary repositories.
Switch to repo_git_path which works on arbitrary repositories.
Signed-off-by: Stefan Beller <redacted>
Signed-off-by: Junio C Hamano <redacted>
---
path.h | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
From: Stefan Beller <hidden> Date: 2018-12-15 00:10:43
dade47c06c (commit-graph: add repo arg to graph readers, 2018-07-11)
brought more independence from the_repository to the commit graph, however
it was not completely independent of the_repository, as the previous
patches show.
To ensure we're not accessing the_repository by accident, we'd ideally
assign NULL to the_repository to trigger a segfault on access.
We currently have a temporary hack in cache.h, which relies on
the_hash_algo (which is a short form of the_repository->hash_algo) to
be set, so we cannot do that. The next best thing is to set all fields of
the_repository to 0, so any accidental access is more likely to be found.
Signed-off-by: Stefan Beller <redacted>
Signed-off-by: Junio C Hamano <redacted>
---
cache.h | 2 ++
t/helper/test-repository.c | 10 ++++++++++
2 files changed, 12 insertions(+)
I realized next has not been rewound, so I can resend sb/more-repo-in-api,
which I hereby do. The changes are minimal and address the only comment
by Jonathan so far.
Sorry I'm very late to look at this, but your series looks good to me.
I've got some work on the way that will use these arbitrary
repositories. (Specifically, moving 'generation' out of 'struct commit'
and into a commit slab to store reachability index values.)
Thanks,
-Stolee
From: Junio C Hamano <hidden> Date: 2018-12-28 20:12:21
Stefan Beller [off-list ref] writes:
I realized next has not been rewound, so I can resend sb/more-repo-in-api,
which I hereby do. The changes are minimal and address the only comment
by Jonathan so far.
Yeah, the only change I see matches what is in your range-diff after
applying them to my tree.
Will rewind 'next' by the end of the year and replace the topic. Thanks.
1: 99017ffac8 ! 1: f24b120287 submodule: use submodule repos for object lookup
@@ -40,12 +40,13 @@
- * attempt to lookup both the left and right commits and put them into the
- * left and right pointers.
+/*
-+ * Initialize 'out' based on the provided submodule path.
++ * Initialize a repository struct for a submodule based on the provided 'path'.
+ *
+ * Unlike repo_submodule_init, this tolerates submodules not present
+ * in .gitmodules. This function exists only to preserve historical behavior,
+ *
-+ * Returns 0 on success, -1 when the submodule is not present.
++ * Returns the repository struct on success,
++ * NULL when the submodule is not present.
*/
-static void show_submodule_header(struct diff_options *o, const char *path,
+static struct repository *open_submodule(const char *path)
@@ -59,6 +60,7 @@
+ return NULL;
+ }
+
++ /* Mark it as a submodule */
+ out->submodule_prefix = xstrdup(path);
+
+ strbuf_release(&sb);
2: 809765861c = 2: 25190d6174 submodule: don't add submodule as odb for push
3: 4a7735da72 = 3: 965421aab2 commit-graph: convert remaining functions to handle any repo
4: aeeb1ba49e = 4: bf31f32723 commit: prepare free_commit_buffer and release_commit_memory for any repo
5: 5ffebe9463 = 5: c4e54e6b0d path.h: make REPO_GIT_PATH_FUNC repository agnostic
6: 9c89920c46 = 6: a7ed0c57ba t/helper/test-repository: celebrate independence from the_repository