Re: Rss produced by git is not valid xml?

Kay Sievers [off-list ref] writes:

On Sat, Nov 19, 2005 at 09:52:34AM -0800, Linus Torvalds wrote:

quoted

On Sat, 19 Nov 2005, Junio C Hamano wrote:

quoted

Well, some people on the list seem to think UTF-8 is the one and
only right encoding, so for them if the message does not
identify what it is in, assuming UTF-8 and not doing any
conversion is probably the right thing ;-).

If you replace "assume" with "verify", then I agree.

One problem I have that approach is what to do if it does not
verify.  Reject and ask them to re-run the program with another
option --binary-log-message?

I found some test code I did a while ago for validation of
filesystem labels, cause D-BUS diconnects your session, if you
send an invalid utf-8 string to the bus. :)

Kay

Thanks.  I take it that you are licensing this code to use in
git when we doing what Linus suggests?

On Sat, 26 Nov 2005, Junio C Hamano wrote:

Kay Sievers [off-list ref] writes:

quoted

quoted

If you replace "assume" with "verify", then I agree.

One problem I have that approach is what to do if it does not
verify.  Reject and ask them to re-run the program with another
option --binary-log-message?

We could do that. With perhaps an option to just do the trivial 
"latin1->utf8" translation, which will be correct in most of the western 
world (and, perhaps more importantly - the places it won't be correct in 
will almost universally have an explicit locale setting or similar, since 
otherwise nothing would work).

In other words, in the absense of locale settings, we can pretty much 
assume any 8-bit data is latin1 if it isn't already utf-8. That's what a 
lot of tools do already (eg, gitk automatically does the right thing, 
exactly because it will assume non-proper utf-8 being in latin1).

I'd suggest that the current "-u" flag do the latin1->utf8 autoconversion, 
and _without_ the "-u" flag, you'd just commit it as binary data..

		Linus

On Sat, Nov 26, 2005 at 07:57:48PM -0800, Junio C Hamano wrote:

Kay Sievers [off-list ref] writes:

quoted

On Sat, Nov 19, 2005 at 09:52:34AM -0800, Linus Torvalds wrote:

quoted

On Sat, 19 Nov 2005, Junio C Hamano wrote:

quoted

Well, some people on the list seem to think UTF-8 is the one and
only right encoding, so for them if the message does not
identify what it is in, assuming UTF-8 and not doing any
conversion is probably the right thing ;-).

If you replace "assume" with "verify", then I agree.

quoted

I found some test code I did a while ago for validation of
filesystem labels, cause D-BUS diconnects your session, if you
send an invalid utf-8 string to the bus. :)

Thanks.  I take it that you are licensing this code to use in
git when we doing what Linus suggests?

Sure, it's free to use under any version of the GPL git uses itself.

Thanks,
Kay

When the message body does not identify what encoding it is in,
-u assumes it is in latin-1 and converts it to utf8, which is
the recommended encoding for git commit log messages.

With -u=<encoding>, the conversion is made into the specified
one, instead of utf8, to allow project-local policies.

Signed-off-by: Junio C Hamano <redacted>

---

 * This says [2/3] but does not use the first one in the series,
   to keep mailinfo less dependent on git.  [3/3] integrates it
   to git a bit further by using the configuration file.

 mailinfo.c |   59 +++++++++++++++++++++++++++++++++++------------------------
 1 files changed, 35 insertions(+), 24 deletions(-)

applies-to: dfac5ab58034e7129ba0d8096ca2bb6857df2242
650e4be59b9f385f56e5829d97d09e8440f174b8

diff --git a/mailinfo.c b/mailinfo.c
index cb853df..6d8c933 100644
--- a/mailinfo.c
+++ b/mailinfo.c

@@ -16,7 +16,7 @@ extern char *gitstrcasestr(const char *h
 static FILE *cmitmsg, *patchfile;
 
 static int keep_subject = 0;
-static int metainfo_utf8 = 0;
+static char *metainfo_charset = NULL;
 static char line[1000];
 static char date[1000];
 static char name[1000];

@@ -441,29 +441,38 @@ static int decode_b_segment(char *in, ch
 
 static void convert_to_utf8(char *line, char *charset)
 {
-	if (*charset) {
-		char *in, *out;
-		size_t insize, outsize, nrc;
-		char outbuf[4096]; /* cheat */
-		iconv_t conv = iconv_open("utf-8", charset);
-
-		if (conv == (iconv_t) -1) {
-			fprintf(stderr, "cannot convert from %s to utf-8\n",
-				charset);
+	char *in, *out;
+	size_t insize, outsize, nrc;
+	char outbuf[4096]; /* cheat */
+	static char latin_one[] = "latin-1";
+	char *input_charset = *charset ? charset : latin_one;
+	iconv_t conv = iconv_open(metainfo_charset, input_charset);
+
+	if (conv == (iconv_t) -1) {
+		static int warned_latin1_once = 0;
+		if (input_charset != latin_one) {
+			fprintf(stderr, "cannot convert from %s to %s\n",
+				input_charset, metainfo_charset);
 			*charset = 0;
-			return;
 		}
-		in = line;
-		insize = strlen(in);
-		out = outbuf;
-		outsize = sizeof(outbuf);
-		nrc = iconv(conv, &in, &insize, &out, &outsize);
-		iconv_close(conv);
-		if (nrc == (size_t) -1)
-			return;
-		*out = 0;
-		strcpy(line, outbuf);
+		else if (!warned_latin1_once) {
+			warned_latin1_once = 1;
+			fprintf(stderr, "tried to convert from %s to %s, "
+				"but your iconv does not work with it.\n",
+				input_charset, metainfo_charset);
+		}
+		return;
 	}
+	in = line;
+	insize = strlen(in);
+	out = outbuf;
+	outsize = sizeof(outbuf);
+	nrc = iconv(conv, &in, &insize, &out, &outsize);
+	iconv_close(conv);
+	if (nrc == (size_t) -1)
+		return;
+	*out = 0;
+	strcpy(line, outbuf);
 }
 
 static void decode_header_bq(char *it)

@@ -511,7 +520,7 @@ static void decode_header_bq(char *it)
 		}
 		if (sz < 0)
 			return;
-		if (metainfo_utf8)
+		if (metainfo_charset)
 			convert_to_utf8(piecebuf, charset_q);
 		strcpy(out, piecebuf);
 		out += strlen(out);

@@ -590,7 +599,7 @@ static int handle_commit_msg(void)
 		 * normalize the log message to UTF-8.
 		 */
 		decode_transfer_encoding(line);
-		if (metainfo_utf8)
+		if (metainfo_charset)
 			convert_to_utf8(line, charset);
 		fputs(line, cmitmsg);
 	} while (fgets(line, sizeof(line), stdin) != NULL);

@@ -720,7 +729,9 @@ int main(int argc, char **argv)
 		if (!strcmp(argv[1], "-k"))
 			keep_subject = 1;
 		else if (!strcmp(argv[1], "-u"))
-			metainfo_utf8 = 1;
+			metainfo_charset = "utf-8";
+		else if (!strncmp(argv[1], "-u=", 3))
+			metainfo_charset = argv[1] + 3;
 		else
 			usage();
 		argc--; argv++;

---
0.99.9.GIT

Junio C Hamano wrote:

When the message body does not identify what encoding it is in,
-u assumes it is in latin-1 and converts it to utf8, which is
the recommended encoding for git commit log messages.

With -u=<encoding>, the conversion is made into the specified
one, instead of utf8, to allow project-local policies.

Signed-off-by: Junio C Hamano <redacted>

-u= is very odd syntax.  Typically you see "-u argument" (sometimes you 
have "-u" and "-U argument" as a pair); --foo=argument is used for long 
options, although even there "--foo argument" tends to be used at least 
when the argument is required.

Incidentally, any reason we're not using getopt_long() for command-line 
parsing?

	-hpa