Re: [PATCH v2 5/5] convert: add filter.<driver>.process option

On 27 Jul 2016, at 11:41, Eric Wong [off-list ref] wrote:

larsxschneider@gmail.com wrote:

quoted

+static off_t multi_packet_read(struct strbuf *sb, const int fd, const size_t size)

I'm no expert in C, but this might be const-correctness taken
too far.  I think basing this on the read(2) prototype is less
surprising:

  static ssize_t multi_packet_read(int fd, struct strbuf *sb, size_t size)

Hm... ok. I like `const` because I think it is usually easier to read/understand
functions that do not change their input variables. This way I can communicate
my intention to future people modifying this function!

If this is frowned upon in the Git community then I will add a comment to the
CodingGuidelines and remove the const :)

I agree with your reordering of the parameters, though!

Speaking of coding style... convert.c is already big and gets only bigger 
with this patch (1720 lines). Would it make sense to add a new file 
"convert-pipe-protocol.c"
or something for my additions?

Also what Jeff said about off_t vs size_t, but my previous
emails may have confused you w.r.t. off_t usage...

quoted

+static int multi_packet_write(const char *src, size_t len, const int in, const int out)

Same comment about over const ints above.
len can probably be off_t based on what is below; but you need
to process the loop in ssize_t-friendly chunks.

I think I would prefer to keep it an size_t because that is the
type we get from Git initially. The code will be more clear in v3.

quoted

+{
+	int ret = 1;
+	char header[4];
+	char buffer[8192];
+	off_t bytes_to_write;

What Jeff said, this should be ssize_t to match read(2) and xread

Agreed.

quoted

+	while (ret) {
+		if (in >= 0) {
+			bytes_to_write = xread(in, buffer, sizeof(buffer));
+			if (bytes_to_write < 0)
+				ret &= 0;
+			src = buffer;
+		} else {
+			bytes_to_write = len > LARGE_PACKET_MAX - 4 ? LARGE_PACKET_MAX - 4 : len;
+			len -= bytes_to_write;
+		}
+		if (!bytes_to_write)
+			break;

The whole ret &= .. style error handling is hard-to-follow and
here, a source of bugs.  I think the expected convention on
hitting errors is:

	1) stop whatever you're doing
	2) cleanup
	3) propagate the error to callers

"goto" is an acceptable way of accomplishing this.

For example, byte_to_write may still be negative at this point
(and interpreted as a really big number when cast to unsigned
size_t) and src/buffer could be stack garbage.

I changed the implementation here so that the &= style
is not necessary anymore. However, I will look into "goto"
for the other areas!

quoted

+		set_packet_header(header, bytes_to_write + 4);
+		ret &= write_in_full(out, &header, sizeof(header)) == sizeof(header);
+		ret &= write_in_full(out, src, bytes_to_write) == bytes_to_write;
+	}
+	ret &= write_in_full(out, "0000", 4) == 4;
+	return ret;
+}
+

quoted

+static int apply_protocol_filter(const char *path, const char *src, size_t len,
+						int fd, struct strbuf *dst, const char *cmd,
+						const char *filter_type)
+{

<snip>

quoted

+	if (fd >= 0 && !src) {
+		ret &= fstat(fd, &file_stat) != -1;
+		len = file_stat.st_size;

Same truncation bug I noticed earlier; what I originally meant
is the `len' arg probably ought to be off_t, here, not size_t.
32-bit x86 Linux systems have 32-bit size_t (unsigned), but
large file support means off_t is 64-bits (signed).

OK. Would it be OK to keep size_t for this patch series?

Also, is it worth continuing this function if fstat fails?

No :-)

quoted

+	}
+
+	sigchain_push(SIGPIPE, SIG_IGN);
+
+	packet_write(process->in, "%s\n", filter_type);
+	packet_write(process->in, "%s\n", path);
+	packet_write(process->in, "%zu\n", len);

I'm not sure if "%zu" is portable since we don't do C99 (yet?)
For 64-bit signed off_t, you can probably do:

	packet_write(process->in, "%"PRIuMAX"\n", (uintmax_t)len);

Since we don't have PRIiMAX or intmax_t, here, and a negative
len would be a bug (probably from failed fstat) anyways.

OK. "%zu" is not used in the entire code base. I will go with
your suggestion!

quoted

+	ret &= multi_packet_write(src, len, fd, process->in);

multi_packet_write will probably fail if fstat failed above...

Yes. The error handling is bogus... I thought bitwise "and" would
act the same way as logical "and" (a bit embarrassing to admit that...).

quoted

+	strbuf = packet_read_line(process->out, NULL);

And this may just block or timeout if multi_packet_write failed.

True, but unless there is anything easy to do I would leave that.
Or do you think it is really necessary to introduce "select" and
friends?


Thanks a lot for your review,
Lars