[RFH] two and half potential fixlets to the...

[RFH] two and half potential fixlets to the in-core index handling

From: Junio C Hamano <hidden>
Date: 2016-06-15 22:45:13

While debugging merge-in-C with Miklos, I noticed that relatively recent
codepaths that deal with the in-core index are subtly broken when a single
process calls read_cache() more than once.

A historical invariant in the API is "if you have read the index or
populated it, then the next read_cache() is a no-op and you need to run
discard_cache() beforehand if you want to read a new on-disk index."  I
think this was broken by relatively recent unpack-trees rewrite, namely
34110cd (Make 'unpack_trees()' have a separate source and destination
index, 2008-03-06) and also 7a51ed6 (Make on-disk index representation
separate from in-core one, 2008-01-14).

With the current code, unpack_trees() builds its merge into a cleaned
o->result (an index_state structure), and it is copied to o->dst_index,
which often is the_index.  But the logic in read_from_index() that
implements the historical invariant checks if istate->alloc is NULL (in
which case the istate is empty -- just after startup, or just after
discard_index()).  The "result" index_state does not have alloc because it
is built from scratch and never allocated anything.

This wouldn't work very well.

A hacky solution I have in the attached patch is to waste an xmalloc(1)
and store it there when o->result is created, and also make
read_from_index() pay attention to the cache_nr and the cache_changed
bit. I think it is the safest and minimum fix.

Another independent issue is that istate has name_hash_initialized bit
that records the validity of the name_hash.  discard_index() frees the
name_hash, but it never resets the bit to zero.  I am not sure what the
ramifications of not doing so, but it certainly feels wrong.

Also, discard_index() did not free the array of pointers istate->cache[];
I think it should.

 read-cache.c   |   11 ++++-------
 unpack-trees.c |    2 ++
 2 files changed, 6 insertions(+), 7 deletions(-)

diff --git c/read-cache.c w/read-cache.c
index 2c03ec3..b29f263 100644
--- c/read-cache.c
+++ w/read-cache.c

@@ -1144,7 +1144,7 @@ static inline size_t estimate_cache_size(size_t ondisk_size, unsigned int entrie
 	return ondisk_size + entries*per_entry;
 }
 
-/* remember to discard_cache() before reading a different cache! */
+/* remember to discard_index() before reading a different cache! */
 int read_index_from(struct index_state *istate, const char *path)
 {
 	int fd, i;

@@ -1155,7 +1155,7 @@ int read_index_from(struct index_state *istate, const char *path)
 	size_t mmap_size;
 
 	errno = EBUSY;
-	if (istate->alloc)
+	if (istate->alloc || istate->cache_nr || istate->cache_changed)
 		return istate->cache_nr;
 
 	errno = ENOENT;

@@ -1240,15 +1240,12 @@ unmap:
 
 int discard_index(struct index_state *istate)
 {
-	istate->cache_nr = 0;
-	istate->cache_changed = 0;
-	istate->timestamp = 0;
 	free_hash(&istate->name_hash);
 	cache_tree_free(&(istate->cache_tree));
 	free(istate->alloc);
-	istate->alloc = NULL;
+	free(istate->cache);
 
-	/* no need to throw away allocated active_cache */
+	memset(istate, 0, sizeof(*istate));
 	return 0;
 }

diff --git c/unpack-trees.c w/unpack-trees.c
index cba0aca..016fd46 100644
--- c/unpack-trees.c
+++ w/unpack-trees.c

@@ -378,6 +378,8 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options
 	memset(&o->result, 0, sizeof(o->result));
 	if (o->src_index)
 		o->result.timestamp = o->src_index->timestamp;
+	if (o->dst_index)
+		o->result.alloc = xmalloc(1);
 	o->merge_size = len;
 
 	if (!dfc)

Re: [RFH] two and half potential fixlets to the in-core index handling

From: Linus Torvalds <torvalds@linux-foundation.org>
Date: 2016-06-15 22:45:13


On Sat, 23 Aug 2008, Junio C Hamano wrote:

A hacky solution I have in the attached patch is to waste an xmalloc(1)
and store it there when o->result is created, and also make
read_from_index() pay attention to the cache_nr and the cache_changed
bit. I think it is the safest and minimum fix.

Hmm. Wouldn't it be nicer to just add another bit to istate? We have the 
space already, since we already have a bitfield there, with just one bit 
used?

Something like this..

Untested. Of course.

			Linus

---
 cache.h        |    3 ++-
 read-cache.c   |    4 +++-
 unpack-trees.c |    1 +
 3 files changed, 6 insertions(+), 2 deletions(-)

diff --git a/cache.h b/cache.h
index 68ce6e6..6e41ec4 100644
--- a/cache.h
+++ b/cache.h

@@ -222,7 +222,8 @@ struct index_state {
 	struct cache_tree *cache_tree;
 	time_t timestamp;
 	void *alloc;
-	unsigned name_hash_initialized : 1;
+	unsigned name_hash_initialized : 1,
+		 initialized : 1;
 	struct hash_table name_hash;
 };

diff --git a/read-cache.c b/read-cache.c
index 2c03ec3..35fec46 100644
--- a/read-cache.c
+++ b/read-cache.c

@@ -1155,7 +1155,7 @@ int read_index_from(struct index_state *istate, const char *path)
 	size_t mmap_size;
 
 	errno = EBUSY;
-	if (istate->alloc)
+	if (istate->initialized)
 		return istate->cache_nr;
 
 	errno = ENOENT;

@@ -1195,6 +1195,7 @@ int read_index_from(struct index_state *istate, const char *path)
 	 * index size
 	 */
 	istate->alloc = xmalloc(estimate_cache_size(mmap_size, istate->cache_nr));
+	istate->initialized = 1;
 
 	src_offset = sizeof(*hdr);
 	dst_offset = 0;

@@ -1247,6 +1248,7 @@ int discard_index(struct index_state *istate)
 	cache_tree_free(&(istate->cache_tree));
 	free(istate->alloc);
 	istate->alloc = NULL;
+	istate->initialized = 0;
 
 	/* no need to throw away allocated active_cache */
 	return 0;

diff --git a/unpack-trees.c b/unpack-trees.c
index cba0aca..ef21c62 100644
--- a/unpack-trees.c
+++ b/unpack-trees.c

@@ -376,6 +376,7 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options
 	state.refresh_cache = 1;
 
 	memset(&o->result, 0, sizeof(o->result));
+	o->result.initialized = 1;
 	if (o->src_index)
 		o->result.timestamp = o->src_index->timestamp;
 	o->merge_size = len;

`h`	back out one level
`j`	next message in thread
`k`	previous message in thread
`l`	drill in
`Esc`	close help / fold thread tree
`?`	toggle this help