Re: [PATCH 4/4] git-compat-util.h: dietlibc-friendly x{malloc,realloc,calloc}

Subsystems: the rest

4 messages, 3 authors, 2016-06-15 · open the first message on its own page

Re: [PATCH 4/4] git-compat-util.h: dietlibc-friendly x{malloc,realloc,calloc}

From: Junio C Hamano <hidden>
Date: 2016-06-15 22:42:15

Eric Wong [off-list ref] writes:
dietlibc versions of these allocators returns NULL if a size of zero is
specified.  Obviously, this is a problem with the x* wrappers since
we check for them returning NULL.

Down the line, it'd be better to hunt down and eliminate all calls to
these functions when they are called with a zero argument.  I've already
added some checks for these cases that were exposed by tests.
I agree that it is a bug to rely on *alloc(0) not returning
NULL, but this patch is too risky.  It would be a good thing to
have debugging variant of x* wrappers that barf on a 0-byte
allocation request to find the offending callers, instead of
returning NULL, maybe like the attached patch.

Since eradicating *alloc(0) calls is the right way to go, but it
takes time.  Touching x* wrappers for general public should not
be done before it is done.  It breaks things for everybody,
while the current code is broken only for diet people and
developers that use the debugging variant of x* wrappers.

-- >8 --
diff --git a/git-compat-util.h b/git-compat-util.h
index 0c98c99..08fd6bc 100644
--- a/git-compat-util.h
+++ b/git-compat-util.h
@@ -60,9 +60,17 @@ extern int gitsetenv(const char *, const
 extern char *gitstrcasestr(const char *haystack, const char *needle);
 #endif
 
+#ifdef DEBUG_0ALLOC
+#define debug_0alloc(sz) assert(0 < (sz))
+#else
+#define debug_0alloc(sz)
+#endif
+
 static inline void *xmalloc(size_t size)
 {
 	void *ret = malloc(size);
+
+	debug_0alloc(size);
 	if (!ret)
 		die("Out of memory, malloc failed");
 	return ret;
@@ -71,6 +79,7 @@ static inline void *xmalloc(size_t size)
 static inline void *xrealloc(void *ptr, size_t size)
 {
 	void *ret = realloc(ptr, size);
+	debug_0alloc(size);
 	if (!ret)
 		die("Out of memory, realloc failed");
 	return ret;
@@ -79,6 +88,7 @@ static inline void *xrealloc(void *ptr, 
 static inline void *xcalloc(size_t nmemb, size_t size)
 {
 	void *ret = calloc(nmemb, size);
+	debug_0alloc(nmemb);
 	if (!ret)
 		die("Out of memory, calloc failed");
 	return ret;

Re: [PATCH 4/4] git-compat-util.h: dietlibc-friendly x{malloc,realloc,calloc}

From: Eric Wong <hidden>
Date: 2016-06-15 22:42:15

Junio C Hamano [off-list ref] wrote:
Eric Wong [off-list ref] writes:
quoted
dietlibc versions of these allocators returns NULL if a size of zero is
specified.  Obviously, this is a problem with the x* wrappers since
we check for them returning NULL.

Down the line, it'd be better to hunt down and eliminate all calls to
these functions when they are called with a zero argument.  I've already
added some checks for these cases that were exposed by tests.
I agree that it is a bug to rely on *alloc(0) not returning
NULL, but this patch is too risky.  It would be a good thing to
have debugging variant of x* wrappers that barf on a 0-byte
allocation request to find the offending callers, instead of
returning NULL, maybe like the attached patch.

Since eradicating *alloc(0) calls is the right way to go, but it
takes time.  Touching x* wrappers for general public should not
be done before it is done.  It breaks things for everybody,
while the current code is broken only for diet people and
developers that use the debugging variant of x* wrappers.
I'll be using the following patch to collect results for the next few
days without interrupting my work on other projects.  I'll post
patches to remove *alloc(0) calls when I find them.
diff --git a/git-compat-util.h b/git-compat-util.h
index 0c98c99..43be289 100644
--- a/git-compat-util.h
+++ b/git-compat-util.h
@@ -60,9 +60,41 @@ extern int gitsetenv(const char *, const
 extern char *gitstrcasestr(const char *haystack, const char *needle);
 #endif
 
+static void fork_for_core (const char * caller)
+{
+	pid_t child = fork();
+	if (child == 0) {
+		int i;
+		char out[4096];
+		char * progname = "unknown";
+		FILE * dbg = fopen("/var/tmp/git_alloc_dbg.log","a");
+
+		for(i = 0; environ[i]; i++) {
+			if (strstr(environ[i],"_=") == environ[i])
+				progname = environ[i];
+		}
+		
+		snprintf(out, 4096, "%s:%s: pid %d dumping core for pid %d\n",
+				progname, caller, getpid(), getppid());
+
+		fputs(out, stderr);
+		fputs(out, dbg);
+		fflush(NULL);
+		
+		abort();
+	}
+}
+
 static inline void *xmalloc(size_t size)
 {
-	void *ret = malloc(size);
+	void *ret;
+	
+	if (!size) {
+		fork_for_core(__func__);
+		return NULL;
+	}
+	
+	ret = malloc(size);
 	if (!ret)
 		die("Out of memory, malloc failed");
 	return ret;
@@ -70,7 +102,15 @@ static inline void *xmalloc(size_t size)
 
 static inline void *xrealloc(void *ptr, size_t size)
 {
-	void *ret = realloc(ptr, size);
+	void *ret;
+	
+	if (!size) {
+		fork_for_core(__func__);
+		free(ptr);
+		return NULL;
+	}
+	
+	ret = realloc(ptr, size);
 	if (!ret)
 		die("Out of memory, realloc failed");
 	return ret;
@@ -78,7 +118,14 @@ static inline void *xrealloc(void *ptr, 
 
 static inline void *xcalloc(size_t nmemb, size_t size)
 {
-	void *ret = calloc(nmemb, size);
+	void *ret;
+	
+	if (!nmemb || !size) {
+		fork_for_core(__func__);
+		return NULL;
+	}
+
+	ret = calloc(nmemb, size);
 	if (!ret)
 		die("Out of memory, calloc failed");
 	return ret;
-- 
Eric Wong

[PATCH] Avoid allocating 0 bytes, was Re: [PATCH 4/4] git-compat-util.h: dietlibc-friendly x{malloc,realloc,calloc}

From: Johannes Schindelin <hidden>
Date: 2016-06-15 22:42:15

This is the result of a relatively quick audit of the source code. There
might still be a few odd places lurking out there, but I am quite certain
I caught most if not all.

Signed-off-by: Johannes Schindelin <redacted>

---

 csum-file.c         |    3 +++
 diff.c              |    6 +++--
 diffcore-order.c    |    8 ++++++-
 diffcore-pathspec.c |   12 +++++++----
 index-pack.c        |   11 ++++++----
 pack-objects.c      |    6 ++++-
 read-tree.c         |   17 +++++++++------
 sha1_file.c         |   56 ++++++++++++++++++++++++++-------------------------
 tree-diff.c         |    4 ++++
 unpack-objects.c    |    2 +-
 10 files changed, 76 insertions(+), 49 deletions(-)

751737aeec4dd3cc1afd1d1d3fa529cfc74535e0
diff --git a/csum-file.c b/csum-file.c
index 5f9249a..2c0f097 100644
--- a/csum-file.c
+++ b/csum-file.c
@@ -121,6 +121,9 @@ int sha1write_compressed(struct sha1file
 	unsigned long maxsize;
 	void *out;
 
+	if (size == 0)
+		return 0;
+
 	memset(&stream, 0, sizeof(stream));
 	deflateInit(&stream, Z_DEFAULT_COMPRESSION);
 	maxsize = deflateBound(&stream, size);
diff --git a/diff.c b/diff.c
index c815918..daee05b 100644
--- a/diff.c
+++ b/diff.c
@@ -504,9 +504,9 @@ static void prepare_temp_file(const char
 		}
 		if (S_ISLNK(st.st_mode)) {
 			int ret;
-			char *buf, buf_[1024];
-			buf = ((sizeof(buf_) < st.st_size) ?
-			       xmalloc(st.st_size) : buf_);
+			char buf[MAXPATHLEN + 1];
+			if (st.st_size > MAXPATHLEN)
+				die("Unbelievably long symlink: %", name);
 			ret = readlink(name, buf, st.st_size);
 			if (ret < 0)
 				die("readlink(%s)", name);
diff --git a/diffcore-order.c b/diffcore-order.c
index b381223..94a4996 100644
--- a/diffcore-order.c
+++ b/diffcore-order.c
@@ -61,6 +61,8 @@ static void prepare_order(const char *or
 		}
 		if (pass == 0) {
 			order_cnt = cnt;
+			if (cnt == 0)
+				return;
 			order = xmalloc(sizeof(*order) * cnt);
 		}
 	}
@@ -105,9 +107,13 @@ static int compare_pair_order(const void
 void diffcore_order(const char *orderfile)
 {
 	struct diff_queue_struct *q = &diff_queued_diff;
-	struct pair_order *o = xmalloc(sizeof(*o) * q->nr);
+	struct pair_order *o;
 	int i;
 
+	if (q->nr == 0)
+		return;
+
+	o = xmalloc(sizeof(*o) * q->nr);
 	prepare_order(orderfile);
 	for (i = 0; i < q->nr; i++) {
 		o[i].pair = q->queue[i];
diff --git a/diffcore-pathspec.c b/diffcore-pathspec.c
index 68fe009..a12337a 100644
--- a/diffcore-pathspec.c
+++ b/diffcore-pathspec.c
@@ -48,10 +48,14 @@ void diffcore_pathspec(const char **path
 	for (i = 0; pathspec[i]; i++)
 		;
 	speccnt = i;
-	spec = xmalloc(sizeof(*spec) * speccnt);
-	for (i = 0; pathspec[i]; i++) {
-		spec[i].spec = pathspec[i];
-		spec[i].len = strlen(pathspec[i]);
+	if (speccnt == 0)
+		spec = NULL;
+	else {
+		spec = xmalloc(sizeof(*spec) * speccnt);
+		for (i = 0; pathspec[i]; i++) {
+			spec[i].spec = pathspec[i];
+			spec[i].len = strlen(pathspec[i]);
+		}
 	}
 
 	for (i = 0; i < q->nr; i++) {
diff --git a/index-pack.c b/index-pack.c
index d4ce3af..2927632 100644
--- a/index-pack.c
+++ b/index-pack.c
@@ -103,7 +103,7 @@ static void *unpack_entry_data(unsigned 
 	unsigned long pack_limit = pack_size - 20;
 	unsigned long pos = *current_pos;
 	z_stream stream;
-	void *buf = xmalloc(size);
+	void *buf = xmalloc(size ? size : 1);
 
 	memset(&stream, 0, sizeof(stream));
 	stream.next_out = buf;
@@ -353,7 +353,8 @@ static void write_index_file(const char 
 {
 	struct sha1file *f;
 	struct object_entry **sorted_by_sha =
-		xcalloc(nr_objects, sizeof(struct object_entry *));
+		xcalloc(nr_objects ? nr_objects : 1,
+				sizeof(struct object_entry *));
 	struct object_entry **list = sorted_by_sha;
 	struct object_entry **last = sorted_by_sha + nr_objects;
 	unsigned int array[256];
@@ -448,8 +449,10 @@ int main(int argc, char **argv)
 
 	open_pack_file();
 	parse_pack_header();
-	objects = xcalloc(nr_objects, sizeof(struct object_entry));
-	deltas = xcalloc(nr_objects, sizeof(struct delta_entry));
+	objects = xcalloc(nr_objects ? nr_objects : 1,
+			sizeof(struct object_entry));
+	deltas = xcalloc(nr_objects ? nr_objects : 1,
+			sizeof(struct delta_entry));
 	parse_pack_objects();
 	free(deltas);
 	write_index_file(index_name, sha1);
diff --git a/pack-objects.c b/pack-objects.c
index caf3b6b..f5a147f 100644
--- a/pack-objects.c
+++ b/pack-objects.c
@@ -406,7 +406,8 @@ static void prepare_pack(int window, int
 
 	fprintf(stderr, "Packing %d objects\n", nr_objects);
 
-	sorted_by_type = create_sorted_list(type_size_sort);
+	if (nr_objects > 0)
+		sorted_by_type = create_sorted_list(type_size_sort);
 	if (window && depth)
 		find_deltas(sorted_by_type, window+1, depth);
 	write_pack_file();
@@ -540,7 +541,8 @@ int main(int argc, char **argv)
 	if (non_empty && !nr_objects)
 		return 0;
 
-	sorted_by_sha = create_sorted_list(sha1_sort);
+	if (nr_objects > 0)
+		sorted_by_sha = create_sorted_list(sha1_sort);
 	SHA1_Init(&ctx);
 	list = sorted_by_sha;
 	for (i = 0; i < nr_objects; i++) {
diff --git a/read-tree.c b/read-tree.c
index e3b9c0d..a46c6fe 100644
--- a/read-tree.c
+++ b/read-tree.c
@@ -294,17 +294,20 @@ static int unpack_trees(merge_fn_t fn)
 {
 	int indpos = 0;
 	unsigned len = object_list_length(trees);
-	struct tree_entry_list **posns = 
-		xmalloc(len * sizeof(struct tree_entry_list *));
+	struct tree_entry_list **posns;
 	int i;
 	struct object_list *posn = trees;
 	merge_size = len;
-	for (i = 0; i < len; i++) {
-		posns[i] = ((struct tree *) posn->item)->entries;
-		posn = posn->next;
+
+	if (len) {
+		posns = xmalloc(len * sizeof(struct tree_entry_list *));
+		for (i = 0; i < len; i++) {
+			posns[i] = ((struct tree *) posn->item)->entries;
+			posn = posn->next;
+		}
+		if (unpack_trees_rec(posns, len, "", fn, &indpos))
+			return -1;
 	}
-	if (unpack_trees_rec(posns, len, "", fn, &indpos))
-		return -1;
 
 	if (trivial_merges_only && nontrivial_merge)
 		die("Merge requires file-level merging");
diff --git a/sha1_file.c b/sha1_file.c
index 8bebbb2..0d60884 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -1371,35 +1371,37 @@ int write_sha1_file(void *buf, unsigned 
 		return -1;
 	}
 
-	/* Set it up */
-	memset(&stream, 0, sizeof(stream));
-	deflateInit(&stream, Z_BEST_COMPRESSION);
-	size = deflateBound(&stream, len+hdrlen);
-	compressed = xmalloc(size);
-
-	/* Compress it */
-	stream.next_out = compressed;
-	stream.avail_out = size;
-
-	/* First header.. */
-	stream.next_in = hdr;
-	stream.avail_in = hdrlen;
-	while (deflate(&stream, 0) == Z_OK)
-		/* nothing */;
-
-	/* Then the data itself.. */
-	stream.next_in = buf;
-	stream.avail_in = len;
-	while (deflate(&stream, Z_FINISH) == Z_OK)
-		/* nothing */;
-	deflateEnd(&stream);
-	size = stream.total_out;
+	if (len + hdrlen > 0) {
+		/* Set it up */
+		memset(&stream, 0, sizeof(stream));
+		deflateInit(&stream, Z_BEST_COMPRESSION);
+		size = deflateBound(&stream, len+hdrlen);
+		compressed = xmalloc(size);
+
+		/* Compress it */
+		stream.next_out = compressed;
+		stream.avail_out = size;
 
-	if (write(fd, compressed, size) != size)
-		die("unable to write file");
+		/* First header.. */
+		stream.next_in = hdr;
+		stream.avail_in = hdrlen;
+		while (deflate(&stream, 0) == Z_OK)
+			/* nothing */;
+
+		/* Then the data itself.. */
+		stream.next_in = buf;
+		stream.avail_in = len;
+		while (deflate(&stream, Z_FINISH) == Z_OK)
+			/* nothing */;
+		deflateEnd(&stream);
+		size = stream.total_out;
+
+		if (write(fd, compressed, size) != size)
+			die("unable to write file");
+		free(compressed);
+	}
 	fchmod(fd, 0444);
 	close(fd);
-	free(compressed);
 
 	return move_temp_to_file(tmpfile, filename);
 }
@@ -1429,7 +1431,7 @@ int write_sha1_to_fd(int fd, const unsig
 		memset(&stream, 0, sizeof(stream));
 		deflateInit(&stream, Z_BEST_COMPRESSION);
 		size = deflateBound(&stream, len + hdrlen);
-		temp_obj = buf = xmalloc(size);
+		temp_obj = buf = xmalloc(size ? size : 1);
 
 		/* Compress it */
 		stream.next_out = buf;
diff --git a/tree-diff.c b/tree-diff.c
index 0ef06a9..382092b 100644
--- a/tree-diff.c
+++ b/tree-diff.c
@@ -263,6 +263,10 @@ void diff_tree_setup_paths(const char **
 
 		paths = p;
 		nr_paths = count_paths(paths);
+		if (nr_paths == 0) {
+			pathlens = NULL;
+			return;
+		}
 		pathlens = xmalloc(nr_paths * sizeof(int));
 		for (i=0; i<nr_paths; i++)
 			pathlens[i] = strlen(paths[i]);
diff --git a/unpack-objects.c b/unpack-objects.c
index 5c5cb12..572eeee 100644
--- a/unpack-objects.c
+++ b/unpack-objects.c
@@ -53,7 +53,7 @@ static void use(int bytes)
 static void *get_data(unsigned long size)
 {
 	z_stream stream;
-	void *buf = xmalloc(size);
+	void *buf = xmalloc(size ? size : 1);
 
 	memset(&stream, 0, sizeof(stream));
 
-- 
1.0.GIT

Re: [PATCH] Avoid allocating 0 bytes, was Re: [PATCH 4/4] git-compat-util.h: dietlibc-friendly x{malloc,realloc,calloc}

From: Eric Wong <hidden>
Date: 2016-06-15 22:42:15

Johannes Schindelin [off-list ref] wrote:
quoted hunk
diff --git a/index-pack.c b/index-pack.c
index d4ce3af..2927632 100644
--- a/index-pack.c
+++ b/index-pack.c
@@ -103,7 +103,7 @@ static void *unpack_entry_data(unsigned 
 	unsigned long pack_limit = pack_size - 20;
 	unsigned long pos = *current_pos;
 	z_stream stream;
-	void *buf = xmalloc(size);
+	void *buf = xmalloc(size ? size : 1);
 
 	memset(&stream, 0, sizeof(stream));
 	stream.next_out = buf;
This is the only one I've managed to hit upon further use the past week
of usage.  No way to break out of the function, either, because
current_pos needs to be updated (and it is changed despite a zero-byte
blob object being in my repository).

-- 
Eric Wong
Keyboard shortcuts
hback out one level
jnext message in thread
kprevious message in thread
ldrill in
Escclose help / fold thread tree
?toggle this help