Re: [PATCH] index-pack: fix allocation of sorted_by_pos array

7 messages, 3 authors, 2016-06-15 · open the first message on its own page

Re: [PATCH] index-pack: fix allocation of sorted_by_pos array

From: Junio C Hamano <hidden>
Date: 2016-06-15 23:05:41

Duy Nguyen [off-list ref] writes:
I keep tripping over this "real_type vs type" in this code. What do
you think about renaming "type" field to "in_pack_type" and
"real_type" to "canon_type" (or "final_type")? "Real" does not really
say anything in this context..
An unqualified name "type" does bother me for the word to express
what representation the piece of data uses (i.e. is it a delta, or
is it a base object of "tree" type, or what).  I think I tried to
unconfuse myself by saying "representation type" in in-code
comments, reviews and log messages when it is not clear which kind
between "in-pack representation" or "Git object type of that stored
data" a sentence is talking about, and I agree "in_pack_type" would
be a vast improvement over just "type".

To me personally real- and final- mean about the same thing
(i.e. what is the real type of the object that is stored?) in the
context of this codepath.

Especially, if the other one is renamed with "in_pack_" prefix,
"real_type" is not just clear enough but is probably better because
it explains what it is from its "meaning" (i.e. it is the type of
the Git object, not how it is represented in the pack-stream) than
"final_type" that is named after "how" it is computed (i.e. it makes
sense to you only if you know that an in-pack type "this is delta"
does not have the full information and you have to traverse the
delta chain and you will finally find out what it is when you hit
the base representation).

Thanks.

Re: [PATCH] index-pack: fix allocation of sorted_by_pos array

From: Jeff King <hidden>
Date: 2016-06-15 23:05:41

On Tue, Jul 07, 2015 at 08:49:19AM -0700, Junio C Hamano wrote:
Duy Nguyen [off-list ref] writes:
quoted
I keep tripping over this "real_type vs type" in this code. What do
you think about renaming "type" field to "in_pack_type" and
"real_type" to "canon_type" (or "final_type")? "Real" does not really
say anything in this context..
An unqualified name "type" does bother me for the word to express
what representation the piece of data uses (i.e. is it a delta, or
is it a base object of "tree" type, or what).  I think I tried to
unconfuse myself by saying "representation type" in in-code
comments, reviews and log messages when it is not clear which kind
between "in-pack representation" or "Git object type of that stored
data" a sentence is talking about, and I agree "in_pack_type" would
be a vast improvement over just "type".
I think this is doubly confusing because pack-objects _does_ use
in_pack_type. And its "type" is therefore the "real" object type. Which
is the opposite of index-pack, which uses "type" for the in-pack type.
So at the very least, we should harmonize these two uses.
Especially, if the other one is renamed with "in_pack_" prefix,
"real_type" is not just clear enough but is probably better because
it explains what it is from its "meaning" (i.e. it is the type of
the Git object, not how it is represented in the pack-stream) than
"final_type" that is named after "how" it is computed (i.e. it makes
sense to you only if you know that an in-pack type "this is delta"
does not have the full information and you have to traverse the
delta chain and you will finally find out what it is when you hit
the base representation).
Yeah, I agree real_type is fine when paired with in_pack_type. We might
consider modifying pack-objects.h to match (on top of just moving
index-pack to in_pack_type).

-Peff

[PATCH 2/2] pack-objects: rename the field "type" to "real_type"

From: Nguyễn Thái Ngọc Duy <hidden>
Date: 2016-06-15 23:05:42

This is to avoid the too generic name "type" and harmonize with the
naming in index-pack. There's a subtle difference though: real_type in
index-pack is what the upper level see, no delta types (after delta
resolution). But real_type in pack-objects is the type to be written in
the pack, delta types are fine (it's actually markers for reused deltas)

Signed-off-by: Nguyễn Thái Ngọc Duy <redacted>
---
 builtin/pack-objects.c | 36 ++++++++++++++++++------------------
 pack-bitmap-write.c    |  6 +++---
 pack-objects.h         |  2 +-
 3 files changed, 22 insertions(+), 22 deletions(-)
diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
index 80fe8c7..e03bf3e 100644
--- a/builtin/pack-objects.c
+++ b/builtin/pack-objects.c
@@ -244,7 +244,7 @@ static unsigned long write_no_reuse_object(struct sha1file *f, struct object_ent
 	struct git_istream *st = NULL;
 
 	if (!usable_delta) {
-		if (entry->type == OBJ_BLOB &&
+		if (entry->real_type == OBJ_BLOB &&
 		    entry->size > big_file_threshold &&
 		    (st = open_istream(entry->idx.sha1, &type, &size, NULL)) != NULL)
 			buf = NULL;
@@ -348,7 +348,7 @@ static unsigned long write_reuse_object(struct sha1file *f, struct object_entry
 	struct pack_window *w_curs = NULL;
 	struct revindex_entry *revidx;
 	off_t offset;
-	enum object_type type = entry->type;
+	enum object_type type = entry->real_type;
 	unsigned long datalen;
 	unsigned char header[10], dheader[10];
 	unsigned hdrlen;
@@ -452,11 +452,11 @@ static unsigned long write_object(struct sha1file *f,
 		to_reuse = 0;	/* explicit */
 	else if (!entry->in_pack)
 		to_reuse = 0;	/* can't reuse what we don't have */
-	else if (entry->type == OBJ_REF_DELTA || entry->type == OBJ_OFS_DELTA)
+	else if (entry->real_type == OBJ_REF_DELTA || entry->real_type == OBJ_OFS_DELTA)
 				/* check_object() decided it for us ... */
 		to_reuse = usable_delta;
 				/* ... but pack split may override that */
-	else if (entry->type != entry->in_pack_type)
+	else if (entry->real_type != entry->in_pack_type)
 		to_reuse = 0;	/* pack has delta which is unusable */
 	else if (entry->delta)
 		to_reuse = 0;	/* we want to pack afresh */
@@ -676,8 +676,8 @@ static struct object_entry **compute_write_order(void)
 	 * And then all remaining commits and tags.
 	 */
 	for (i = last_untagged; i < to_pack.nr_objects; i++) {
-		if (objects[i].type != OBJ_COMMIT &&
-		    objects[i].type != OBJ_TAG)
+		if (objects[i].real_type != OBJ_COMMIT &&
+		    objects[i].real_type != OBJ_TAG)
 			continue;
 		add_to_write_order(wo, &wo_end, &objects[i]);
 	}
@@ -686,7 +686,7 @@ static struct object_entry **compute_write_order(void)
 	 * And then all the trees.
 	 */
 	for (i = last_untagged; i < to_pack.nr_objects; i++) {
-		if (objects[i].type != OBJ_TREE)
+		if (objects[i].real_type != OBJ_TREE)
 			continue;
 		add_to_write_order(wo, &wo_end, &objects[i]);
 	}
@@ -994,7 +994,7 @@ static void create_object_entry(const unsigned char *sha1,
 	entry = packlist_alloc(&to_pack, sha1, index_pos);
 	entry->hash = hash;
 	if (type)
-		entry->type = type;
+		entry->real_type = type;
 	if (exclude)
 		entry->preferred_base = 1;
 	else
@@ -1355,9 +1355,9 @@ static void check_object(struct object_entry *entry)
 		switch (entry->in_pack_type) {
 		default:
 			/* Not a delta hence we've already got all we need. */
-			entry->type = entry->in_pack_type;
+			entry->real_type = entry->in_pack_type;
 			entry->in_pack_header_size = used;
-			if (entry->type < OBJ_COMMIT || entry->type > OBJ_BLOB)
+			if (entry->real_type < OBJ_COMMIT || entry->real_type > OBJ_BLOB)
 				goto give_up;
 			unuse_pack(&w_curs);
 			return;
@@ -1411,7 +1411,7 @@ static void check_object(struct object_entry *entry)
 			 * deltify other objects against, in order to avoid
 			 * circular deltas.
 			 */
-			entry->type = entry->in_pack_type;
+			entry->real_type = entry->in_pack_type;
 			entry->delta = base_entry;
 			entry->delta_size = entry->size;
 			entry->delta_sibling = base_entry->delta_child;
@@ -1420,7 +1420,7 @@ static void check_object(struct object_entry *entry)
 			return;
 		}
 
-		if (entry->type) {
+		if (entry->real_type) {
 			/*
 			 * This must be a delta and we already know what the
 			 * final object type is.  Let's extract the actual
@@ -1443,7 +1443,7 @@ static void check_object(struct object_entry *entry)
 		unuse_pack(&w_curs);
 	}
 
-	entry->type = sha1_object_info(entry->idx.sha1, &entry->size);
+	entry->real_type = sha1_object_info(entry->idx.sha1, &entry->size);
 	/*
 	 * The error condition is checked in prepare_pack().  This is
 	 * to permit a missing preferred base object to be ignored
@@ -1503,9 +1503,9 @@ static int type_size_sort(const void *_a, const void *_b)
 	const struct object_entry *a = *(struct object_entry **)_a;
 	const struct object_entry *b = *(struct object_entry **)_b;
 
-	if (a->type > b->type)
+	if (a->real_type > b->real_type)
 		return -1;
-	if (a->type < b->type)
+	if (a->real_type < b->real_type)
 		return 1;
 	if (a->hash > b->hash)
 		return -1;
@@ -1581,7 +1581,7 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,
 	void *delta_buf;
 
 	/* Don't bother doing diffs between different types */
-	if (trg_entry->type != src_entry->type)
+	if (trg_entry->real_type != src_entry->real_type)
 		return -1;
 
 	/*
@@ -2149,11 +2149,11 @@ static void prepare_pack(int window, int depth)
 
 		if (!entry->preferred_base) {
 			nr_deltas++;
-			if (entry->type < 0)
+			if (entry->real_type < 0)
 				die("unable to get type of object %s",
 				    sha1_to_hex(entry->idx.sha1));
 		} else {
-			if (entry->type < 0) {
+			if (entry->real_type < 0) {
 				/*
 				 * This object is not found, but we
 				 * don't have to include it anyway.
diff --git a/pack-bitmap-write.c b/pack-bitmap-write.c
index c05d138..572f4d6 100644
--- a/pack-bitmap-write.c
+++ b/pack-bitmap-write.c
@@ -64,12 +64,12 @@ void bitmap_writer_build_type_index(struct pack_idx_entry **index,
 
 		entry->in_pack_pos = i;
 
-		switch (entry->type) {
+		switch (entry->real_type) {
 		case OBJ_COMMIT:
 		case OBJ_TREE:
 		case OBJ_BLOB:
 		case OBJ_TAG:
-			real_type = entry->type;
+			real_type = entry->real_type;
 			break;
 
 		default:
@@ -96,7 +96,7 @@ void bitmap_writer_build_type_index(struct pack_idx_entry **index,
 
 		default:
 			die("Missing type information for %s (%d/%d)",
-			    sha1_to_hex(entry->idx.sha1), real_type, entry->type);
+			    sha1_to_hex(entry->idx.sha1), real_type, entry->real_type);
 		}
 	}
 }
diff --git a/pack-objects.h b/pack-objects.h
index d1b98b3..33cde59 100644
--- a/pack-objects.h
+++ b/pack-objects.h
@@ -14,7 +14,7 @@ struct object_entry {
 	void *delta_data;	/* cached delta (uncompressed) */
 	unsigned long delta_size;	/* delta data size (uncompressed) */
 	unsigned long z_delta_size;	/* delta data size (compressed) */
-	enum object_type type;
+	enum object_type real_type;
 	enum object_type in_pack_type;	/* could be delta */
 	uint32_t hash;			/* name hint hash */
 	unsigned int in_pack_pos;
-- 
2.3.0.rc1.137.g477eb31

[PATCH 1/2] index-pack: rename the field "type" to "in_pack_type"

From: Nguyễn Thái Ngọc Duy <hidden>
Date: 2016-06-15 23:05:42

We have two types in this code: in-pack and canonical. "in_pack_type"
makes it clearer than plain "type".

Signed-off-by: Nguyễn Thái Ngọc Duy <redacted>
---
 builtin/index-pack.c | 42 +++++++++++++++++++++---------------------
 1 file changed, 21 insertions(+), 21 deletions(-)
diff --git a/builtin/index-pack.c b/builtin/index-pack.c
index 48fa472..797e571 100644
--- a/builtin/index-pack.c
+++ b/builtin/index-pack.c
@@ -19,7 +19,7 @@ struct object_entry {
 	struct pack_idx_entry idx;
 	unsigned long size;
 	unsigned char hdr_size;
-	signed char type;
+	signed char in_pack_type;
 	signed char real_type;
 };
 
@@ -493,7 +493,7 @@ static void *unpack_raw_entry(struct object_entry *obj,
 	p = fill(1);
 	c = *p;
 	use(1);
-	obj->type = (c >> 4) & 7;
+	obj->in_pack_type = (c >> 4) & 7;
 	size = (c & 15);
 	shift = 4;
 	while (c & 0x80) {
@@ -505,7 +505,7 @@ static void *unpack_raw_entry(struct object_entry *obj,
 	}
 	obj->size = size;
 
-	switch (obj->type) {
+	switch (obj->in_pack_type) {
 	case OBJ_REF_DELTA:
 		hashcpy(ref_sha1, fill(20));
 		use(20);
@@ -534,11 +534,11 @@ static void *unpack_raw_entry(struct object_entry *obj,
 	case OBJ_TAG:
 		break;
 	default:
-		bad_object(obj->idx.offset, _("unknown object type %d"), obj->type);
+		bad_object(obj->idx.offset, _("unknown object type %d"), obj->in_pack_type);
 	}
 	obj->hdr_size = consumed_bytes - obj->idx.offset;
 
-	data = unpack_entry_data(obj->idx.offset, obj->size, obj->type, sha1);
+	data = unpack_entry_data(obj->idx.offset, obj->size, obj->in_pack_type, sha1);
 	obj->idx.crc32 = input_crc32;
 	return data;
 }
@@ -631,7 +631,7 @@ static int find_ofs_delta(const off_t offset, enum object_type type)
 		int cmp;
 
 		cmp = compare_ofs_delta_bases(offset, delta->offset,
-					      type, objects[delta->obj_no].type);
+					      type, objects[delta->obj_no].in_pack_type);
 		if (!cmp)
 			return next;
 		if (cmp < 0) {
@@ -685,7 +685,7 @@ static int find_ref_delta(const unsigned char *sha1, enum object_type type)
 		int cmp;
 
 		cmp = compare_ref_delta_bases(sha1, delta->sha1,
-					      type, objects[delta->obj_no].type);
+					      type, objects[delta->obj_no].in_pack_type);
 		if (!cmp)
 			return next;
 		if (cmp < 0) {
@@ -759,7 +759,7 @@ static int check_collison(struct object_entry *entry)
 	enum object_type type;
 	unsigned long size;
 
-	if (entry->size <= big_file_threshold || entry->type != OBJ_BLOB)
+	if (entry->size <= big_file_threshold || entry->in_pack_type != OBJ_BLOB)
 		return -1;
 
 	memset(&data, 0, sizeof(data));
@@ -767,7 +767,7 @@ static int check_collison(struct object_entry *entry)
 	data.st = open_istream(entry->idx.sha1, &type, &size, NULL);
 	if (!data.st)
 		return -1;
-	if (size != entry->size || type != entry->type)
+	if (size != entry->size || type != entry->in_pack_type)
 		die(_("SHA1 COLLISION FOUND WITH %s !"),
 		    sha1_to_hex(entry->idx.sha1));
 	unpack_data(entry, compare_objects, &data);
@@ -891,7 +891,7 @@ static void *get_base_data(struct base_data *c)
 		struct base_data **delta = NULL;
 		int delta_nr = 0, delta_alloc = 0;
 
-		while (is_delta_type(c->obj->type) && !c->data) {
+		while (is_delta_type(c->obj->in_pack_type) && !c->data) {
 			ALLOC_GROW(delta, delta_nr + 1, delta_alloc);
 			delta[delta_nr++] = c;
 			c = c->base;
@@ -1085,7 +1085,7 @@ static void *threaded_second_pass(void *data)
 		counter_unlock();
 		work_lock();
 		while (nr_dispatched < nr_objects &&
-		       is_delta_type(objects[nr_dispatched].type))
+		       is_delta_type(objects[nr_dispatched].in_pack_type))
 			nr_dispatched++;
 		if (nr_dispatched >= nr_objects) {
 			work_unlock();
@@ -1121,12 +1121,12 @@ static void parse_pack_objects(unsigned char *sha1)
 		struct object_entry *obj = &objects[i];
 		void *data = unpack_raw_entry(obj, &ofs_delta->offset,
 					      ref_delta_sha1, obj->idx.sha1);
-		obj->real_type = obj->type;
-		if (obj->type == OBJ_OFS_DELTA) {
+		obj->real_type = obj->in_pack_type;
+		if (obj->in_pack_type == OBJ_OFS_DELTA) {
 			nr_ofs_deltas++;
 			ofs_delta->obj_no = i;
 			ofs_delta++;
-		} else if (obj->type == OBJ_REF_DELTA) {
+		} else if (obj->in_pack_type == OBJ_REF_DELTA) {
 			ALLOC_GROW(ref_deltas, nr_ref_deltas + 1, ref_deltas_alloc);
 			hashcpy(ref_deltas[nr_ref_deltas].sha1, ref_delta_sha1);
 			ref_deltas[nr_ref_deltas].obj_no = i;
@@ -1136,7 +1136,7 @@ static void parse_pack_objects(unsigned char *sha1)
 			obj->real_type = OBJ_BAD;
 			nr_delays++;
 		} else
-			sha1_object(data, NULL, obj->size, obj->type, obj->idx.sha1);
+			sha1_object(data, NULL, obj->size, obj->in_pack_type, obj->idx.sha1);
 		free(data);
 		display_progress(progress, i+1);
 	}
@@ -1161,8 +1161,8 @@ static void parse_pack_objects(unsigned char *sha1)
 		struct object_entry *obj = &objects[i];
 		if (obj->real_type != OBJ_BAD)
 			continue;
-		obj->real_type = obj->type;
-		sha1_object(NULL, obj, obj->size, obj->type, obj->idx.sha1);
+		obj->real_type = obj->in_pack_type;
+		sha1_object(NULL, obj, obj->size, obj->in_pack_type, obj->idx.sha1);
 		nr_delays--;
 	}
 	if (nr_delays)
@@ -1215,7 +1215,7 @@ static void resolve_deltas(void)
 	for (i = 0; i < nr_objects; i++) {
 		struct object_entry *obj = &objects[i];
 
-		if (is_delta_type(obj->type))
+		if (is_delta_type(obj->in_pack_type))
 			continue;
 		resolve_base(obj);
 		display_progress(progress, nr_resolved_deltas);
@@ -1314,7 +1314,7 @@ static struct object_entry *append_obj_to_pack(struct sha1file *f,
 	sha1write(f, header, n);
 	obj[0].size = size;
 	obj[0].hdr_size = n;
-	obj[0].type = type;
+	obj[0].in_pack_type = type;
 	obj[0].real_type = type;
 	obj[1].idx.offset = obj[0].idx.offset + n;
 	obj[1].idx.offset += write_compressed(f, buf, size);
@@ -1566,7 +1566,7 @@ static void show_pack_info(int stat_only)
 	for (i = 0; i < nr_objects; i++) {
 		struct object_entry *obj = &objects[i];
 
-		if (is_delta_type(obj->type))
+		if (is_delta_type(obj->in_pack_type))
 			chain_histogram[obj_stat[i].delta_depth - 1]++;
 		if (stat_only)
 			continue;
@@ -1575,7 +1575,7 @@ static void show_pack_info(int stat_only)
 		       typename(obj->real_type), obj->size,
 		       (unsigned long)(obj[1].idx.offset - obj->idx.offset),
 		       (uintmax_t)obj->idx.offset);
-		if (is_delta_type(obj->type)) {
+		if (is_delta_type(obj->in_pack_type)) {
 			struct object_entry *bobj = &objects[obj_stat[i].base_object_no];
 			printf(" %u %s", obj_stat[i].delta_depth, sha1_to_hex(bobj->idx.sha1));
 		}
-- 
2.3.0.rc1.137.g477eb31

Re: [PATCH 2/2] pack-objects: rename the field "type" to "real_type"

From: Jeff King <hidden>
Date: 2016-06-15 23:05:42

On Wed, Jul 08, 2015 at 06:56:31PM +0700, Nguyễn Thái Ngọc Duy wrote:
This is to avoid the too generic name "type" and harmonize with the
naming in index-pack. There's a subtle difference though: real_type in
index-pack is what the upper level see, no delta types (after delta
resolution). But real_type in pack-objects is the type to be written in
the pack, delta types are fine (it's actually markers for reused deltas)
Hrm, now I'm confused about whether this change is a good idea.

The definition of in_pack_type says:
 	enum object_type in_pack_type;	/* could be delta */
so now I am confused about what exactly "type" (and now "real_type")
means.

-Peff

Re: [PATCH 2/2] pack-objects: rename the field "type" to "real_type"

From: Duy Nguyen <hidden>
Date: 2016-06-15 23:05:42

On Wed, Jul 8, 2015 at 8:47 PM, Jeff King [off-list ref] wrote:
On Wed, Jul 08, 2015 at 06:56:31PM +0700, Nguyễn Thái Ngọc Duy wrote:
quoted
This is to avoid the too generic name "type" and harmonize with the
naming in index-pack. There's a subtle difference though: real_type in
index-pack is what the upper level see, no delta types (after delta
resolution). But real_type in pack-objects is the type to be written in
the pack, delta types are fine (it's actually markers for reused deltas)
Hrm, now I'm confused about whether this change is a good idea.
Oh good :) I found it not-so-good too after seeing the check "if
(real_type == OBJ_REF...)"
The definition of in_pack_type says:
quoted
      enum object_type in_pack_type;  /* could be delta */
so now I am confused about what exactly "type" (and now "real_type")
means.
I think we just overload "type" with "this delta is detected reusable
already" in write_object(). It only means 'real type in the output
pack' for canonical types. For generated deltas, we already know if
it's ref-delta or ofs-delta, we don't rely on real_type
-- 
Duy

Re: [PATCH 2/2] pack-objects: rename the field "type" to "real_type"

From: Jeff King <hidden>
Date: 2016-06-15 23:05:42

On Wed, Jul 08, 2015 at 08:57:35PM +0700, Duy Nguyen wrote:
quoted
The definition of in_pack_type says:
quoted
      enum object_type in_pack_type;  /* could be delta */
so now I am confused about what exactly "type" (and now "real_type")
means.
I think we just overload "type" with "this delta is detected reusable
already" in write_object(). It only means 'real type in the output
pack' for canonical types. For generated deltas, we already know if
it's ref-delta or ofs-delta, we don't rely on real_type
Ah. I think I'd be inclined to just leave it as "type" then.

-Peff
Keyboard shortcuts
hback out one level
jnext message in thread
kprevious message in thread
ldrill in
Escclose help / fold thread tree
?toggle this help