[PATCH v2 0/2] git-p4: handle "Translation of file content failed"

From: Lars Schneider <redacted>

diff to v1:
* add a test case
* use Popen "communication" function instead of "wait"

Thanks to Junio for feedback!

Known issue: My fix works only if git-p4 is executed in verbose mode.
In normal mode no exceptions are thrown and git-p4 just exits.

Lars Schneider (2):
  git-p4: add test case for "Translation of file content failed" error
  git-p4: handle "Translation of file content failed"

 git-p4.py                                  | 27 ++++++++++-------
 t/t9824-git-p4-handle-utf16-without-bom.sh | 47 ++++++++++++++++++++++++++++++
 2 files changed, 63 insertions(+), 11 deletions(-)
 create mode 100755 t/t9824-git-p4-handle-utf16-without-bom.sh

--
2.5.1

From: Lars Schneider <redacted>

A P4 repository can get into a state where it contains a file with
type UTF-16 that does not contain a valid UTF-16 BOM. If git-p4
attempts to retrieve the file then the process crashes with a
"Translation of file content failed" error.

Fix this by detecting this error and retrieving the file as binary
instead. The result in Git is the same.

Known issue: This works only if git-p4 is executed in verbose mode.
In normal mode no exceptions are thrown and git-p4 just exits.

Signed-off-by: Lars Schneider <redacted>
---
 git-p4.py | 27 ++++++++++++++++-----------
 1 file changed, 16 insertions(+), 11 deletions(-)

diff --git a/git-p4.py b/git-p4.py
index 073f87b..5ae25a6 100755
--- a/git-p4.py
+++ b/git-p4.py

@@ -134,13 +134,11 @@ def read_pipe(c, ignore_error=False):
         sys.stderr.write('Reading pipe: %s\n' % str(c))
 
     expand = isinstance(c,basestring)
-    p = subprocess.Popen(c, stdout=subprocess.PIPE, shell=expand)
-    pipe = p.stdout
-    val = pipe.read()
-    if p.wait() and not ignore_error:
-        die('Command failed: %s' % str(c))
-
-    return val
+    p = subprocess.Popen(c, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=expand)
+    (out, err) = p.communicate()
+    if p.returncode != 0 and not ignore_error:
+        die('Command failed: %s\nError: %s' % (str(c), err))
+    return out
 
 def p4_read_pipe(c, ignore_error=False):
     real_cmd = p4_build_cmd(c)

@@ -2186,10 +2184,17 @@ class P4Sync(Command, P4UserMap):
             # them back too.  This is not needed to the cygwin windows version,
             # just the native "NT" type.
             #
-            text = p4_read_pipe(['print', '-q', '-o', '-', "%s@%s" % (file['depotFile'], file['change']) ])
-            if p4_version_string().find("/NT") >= 0:
-                text = text.replace("\r\n", "\n")
-            contents = [ text ]
+            try:
+                text = p4_read_pipe(['print', '-q', '-o', '-', '%s@%s' % (file['depotFile'], file['change'])])
+            except Exception as e:
+                if 'Translation of file content failed' in str(e):
+                    type_base = 'binary'
+                else:
+                    raise e
+            else:
+                if p4_version_string().find('/NT') >= 0:
+                    text = text.replace('\r\n', '\n')
+                contents = [ text ]
 
         if type_base == "apple":
             # Apple filetype files will be streamed as a concatenation of

-- 
2.5.1

From: Lars Schneider <redacted>

A P4 repository can get into a state where it contains a file with
type UTF-16 that does not contain a valid UTF-16 BOM. If git-p4
attempts to retrieve the file then the process crashes with a
"Translation of file content failed" error.

Signed-off-by: Lars Schneider <redacted>
---
 t/t9824-git-p4-handle-utf16-without-bom.sh | 47 ++++++++++++++++++++++++++++++
 1 file changed, 47 insertions(+)
 create mode 100755 t/t9824-git-p4-handle-utf16-without-bom.sh

diff --git a/t/t9824-git-p4-handle-utf16-without-bom.sh b/t/t9824-git-p4-handle-utf16-without-bom.sh
new file mode 100755
index 0000000..fa8043b
--- /dev/null
+++ b/t/t9824-git-p4-handle-utf16-without-bom.sh

@@ -0,0 +1,47 @@
+#!/bin/sh
+
+test_description='git p4 handle UTF-16 without BOM'
+
+. ./lib-git-p4.sh
+
+UTF16="\\x97\\x0\\x97\\x0"
+
+test_expect_success 'start p4d' '
+	start_p4d
+'
+
+test_expect_success 'init depot with UTF-16 encoded file and artificially remove BOM' '
+	(
+		cd "$cli" &&
+		echo "file1 -text" > .gitattributes &&
+		perl -e "printf \"$UTF16\"" >file1 &&
+		p4 add -t utf16 file1 &&
+
+		p4 add .gitattributes &&
+		p4 submit -d "file1"
+	) &&
+
+	(
+		cd "db" &&
+		p4d -jc &&
+		# P4D automatically adds a BOM. Remove it here to make the file invalid.
+		perl -i -ne "print unless eof" depot/file1,v &&
+		perl -e "printf \"@$UTF16@\"" >> depot/file1,v &&
+		p4d -jrF checkpoint.1
+	)
+'
+
+test_expect_success 'clone depot with invalid UTF-16 file' '
+	git p4 clone --dest="$git" --verbose //depot &&
+	(
+		cd "$git" &&
+		perl -e "printf \"$UTF16\"" > expect &&
+		test_cmp_bin expect file1
+	)
+'
+
+test_expect_success 'kill p4d' '
+	kill_p4d
+'
+
+test_done

-- 
2.5.1

On 09/14/2015 06:55 PM, larsxschneider@gmail.com wrote:

Please no space between '>' and the filename,
(this is our coding standard, and the same further down)

+		perl -e "printf \"$UTF16\"" >file1 &&

Ehh, do we need perl here ?
This will invoke a process-fork, which costs time and cpu load.
The following works for me:
printf '\227\000\227\000' >file1

On 14/09/15 17:55, larsxschneider@gmail.com wrote:

From: Lars Schneider <redacted>

A P4 repository can get into a state where it contains a file with
type UTF-16 that does not contain a valid UTF-16 BOM. If git-p4

Sorry - what's a BOM? I'm assuming it's not a Bill of Materials?

Do we know the mechanism by which we end up in this state?

attempts to retrieve the file then the process crashes with a
"Translation of file content failed" error.

Fix this by detecting this error and retrieving the file as binary
instead. The result in Git is the same.

Known issue: This works only if git-p4 is executed in verbose mode.
In normal mode no exceptions are thrown and git-p4 just exits.

Does that mean that the error will only be detected in verbose mode? 
That doesn't seem right!

Would it be better to specify which kind of Exception you are catching? 
Looks like you could get OSError, ValueError and CalledProcessError; 
it's the last of these you want (I think).

+                if 'Translation of file content failed' in str(e):
+                    type_base = 'binary'
+                else:
+                    raise e
+            else:
+                if p4_version_string().find('/NT') >= 0:
+                    text = text.replace('\r\n', '\n')
+                contents = [ text ]

The indentation on this bit doesn't look right to me.

          if type_base == "apple":
              # Apple filetype files will be streamed as a concatenation of

Luke

On 15 Sep 2015, at 06:40, Torsten Bögershausen [off-list ref] wrote:

On 09/14/2015 06:55 PM, larsxschneider@gmail.com wrote:

quoted

From: Lars Schneider <redacted>

A P4 repository can get into a state where it contains a file with
type UTF-16 that does not contain a valid UTF-16 BOM. If git-p4
attempts to retrieve the file then the process crashes with a
"Translation of file content failed" error.

Signed-off-by: Lars Schneider <redacted>
---
 t/t9824-git-p4-handle-utf16-without-bom.sh | 47 ++++++++++++++++++++++++++++++
 1 file changed, 47 insertions(+)
 create mode 100755 t/t9824-git-p4-handle-utf16-without-bom.sh

diff --git a/t/t9824-git-p4-handle-utf16-without-bom.sh b/t/t9824-git-p4-handle-utf16-without-bom.sh
new file mode 100755
index 0000000..fa8043b
--- /dev/null
+++ b/t/t9824-git-p4-handle-utf16-without-bom.sh

@@ -0,0 +1,47 @@
+#!/bin/sh
+
+test_description='git p4 handle UTF-16 without BOM'
+
+. ./lib-git-p4.sh
+
+UTF16="\\x97\\x0\\x97\\x0"
+
+test_expect_success 'start p4d' '
+	start_p4d
+'
+
+test_expect_success 'init depot with UTF-16 encoded file and artificially remove BOM' '
+	(
+		cd "$cli" &&
+		echo "file1 -text" > .gitattributes &&

Please no space between '>' and the filename,
(this is our coding standard, and the same further down)

Correct! Sorry, I still need to get used to this style. Thanks for the reminder!

quoted

+		perl -e "printf \"$UTF16\"" >file1 &&

Ehh, do we need perl here ?
This will invoke a process-fork, which costs time and cpu load.
The following works for me:
printf '\227\000\227\000' >file1

I agree this is better.

Both issues will be fixed v3.

Thanks!

On 15 Sep 2015, at 08:43, Luke Diamand [off-list ref] wrote:

On 14/09/15 17:55, larsxschneider@gmail.com wrote:

quoted

From: Lars Schneider <redacted>

A P4 repository can get into a state where it contains a file with
type UTF-16 that does not contain a valid UTF-16 BOM. If git-p4

Sorry - what's a BOM? I'm assuming it's not a Bill of Materials?

BOM stands for Byte Order Mark. The UTF-16 BOM is a two byte sequence at the beginning of a UTF-16 file. It is not part of the actual content. It is only used to define the encoding of the remaining file. FEFF stands for UTF-16 big-endian encoding and FFFE for little-endian encoding.

More info here: http://www.unicode.org/faq/utf_bom.html#bom1

Do we know the mechanism by which we end up in this state?

Unfortunately no. I tried hard to reproduce the error with “conventional” methods. As you can see I ended up manipulating the P4 database…

However, it looks like this error happens in the wild, too:
https://stackoverflow.com/questions/5156909/translation-of-file-content-failed-error-in-perforce
https://stackoverflow.com/questions/887006/perforce-translation-of-file-content-failed-error

quoted

attempts to retrieve the file then the process crashes with a
"Translation of file content failed" error.

Fix this by detecting this error and retrieving the file as binary
instead. The result in Git is the same.

Known issue: This works only if git-p4 is executed in verbose mode.
In normal mode no exceptions are thrown and git-p4 just exits.

Does that mean that the error will only be detected in verbose mode? That doesn't seem right!

Correct. I don’t like this either but I also don’t want to make huge changes to git-p4.
You can see the root problem here:
https://github.com/git/git/blob/97d7ad75b6fe74960d2a12e4a9151a55a5a87d6d/git-p4.py#L110-L114

Any idea how to approach that best?

quoted

Signed-off-by: Lars Schneider <redacted>
---
 git-p4.py | 27 ++++++++++++++++-----------
 1 file changed, 16 insertions(+), 11 deletions(-)

diff --git a/git-p4.py b/git-p4.py
index 073f87b..5ae25a6 100755
--- a/git-p4.py
+++ b/git-p4.py

@@ -134,13 +134,11 @@ def read_pipe(c, ignore_error=False):
         sys.stderr.write('Reading pipe: %s\n' % str(c))

     expand = isinstance(c,basestring)
-    p = subprocess.Popen(c, stdout=subprocess.PIPE, shell=expand)
-    pipe = p.stdout
-    val = pipe.read()
-    if p.wait() and not ignore_error:
-        die('Command failed: %s' % str(c))
-
-    return val
+    p = subprocess.Popen(c, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=expand)
+    (out, err) = p.communicate()
+    if p.returncode != 0 and not ignore_error:
+        die('Command failed: %s\nError: %s' % (str(c), err))
+    return out

 def p4_read_pipe(c, ignore_error=False):
     real_cmd = p4_build_cmd(c)

@@ -2186,10 +2184,17 @@ class P4Sync(Command, P4UserMap):
             # them back too.  This is not needed to the cygwin windows version,
             # just the native "NT" type.
             #
-            text = p4_read_pipe(['print', '-q', '-o', '-', "%s@%s" % (file['depotFile'], file['change']) ])
-            if p4_version_string().find("/NT") >= 0:
-                text = text.replace("\r\n", "\n")
-            contents = [ text ]
+            try:
+                text = p4_read_pipe(['print', '-q', '-o', '-', '%s@%s' % (file['depotFile'], file['change'])])
+            except Exception as e:

Would it be better to specify which kind of Exception you are catching? Looks like you could get OSError, ValueError and CalledProcessError; it's the last of these you want (I think).

I think it is just a plain exception. See here:
https://github.com/git/git/blob/97d7ad75b6fe74960d2a12e4a9151a55a5a87d6d/git-p4.py#L111

quoted

+                if 'Translation of file content failed' in str(e):
+                    type_base = 'binary'
+                else:
+                    raise e
+            else:
+                if p4_version_string().find('/NT') >= 0:
+                    text = text.replace('\r\n', '\n')
+                contents = [ text ]

The indentation on this bit doesn't look right to me.

I believe it is exactly how it was:
https://github.com/git/git/blob/97d7ad75b6fe74960d2a12e4a9151a55a5a87d6d/git-p4.py#L2397-L2399


In general, what is the appropriate way to reference code in this email list? Are GitHub links OK?


Thanks,
Lars

On 15/09/15 16:38, Lars Schneider wrote:

On 15 Sep 2015, at 08:43, Luke Diamand [off-list ref] wrote:

quoted

Do we know the mechanism by which we end up in this state?

Unfortunately no. I tried hard to reproduce the error with “conventional” methods. As you can see I ended up manipulating the P4 database…

However, it looks like this error happens in the wild, too:
https://stackoverflow.com/questions/5156909/translation-of-file-content-failed-error-in-perforce
https://stackoverflow.com/questions/887006/perforce-translation-of-file-content-failed-error

It's described in the Perforce FAQ here:

http://answers.perforce.com/articles/KB/3117

i.e. it looks to be caused by mixing old and new P4 clients.

quoted

quoted

Known issue: This works only if git-p4 is executed in verbose mode.
In normal mode no exceptions are thrown and git-p4 just exits.

Does that mean that the error will only be detected in verbose mode? That doesn't seem right!

Correct. I don’t like this either but I also don’t want to make huge changes to git-p4.
You can see the root problem here:
https://github.com/git/git/blob/97d7ad75b6fe74960d2a12e4a9151a55a5a87d6d/git-p4.py#L110-L114

Any idea how to approach that best?

I guess what we have is not ideal but probably good enough.

quoted

quoted

+            try:
+                text = p4_read_pipe(['print', '-q', '-o', '-', '%s@%s' % (file['depotFile'], file['change'])])
+            except Exception as e:

Would it be better to specify which kind of Exception you are catching? Looks like you could get OSError, ValueError and CalledProcessError; it's the last of these you want (I think).

I think it is just a plain exception. See here:
https://github.com/git/git/blob/97d7ad75b6fe74960d2a12e4a9151a55a5a87d6d/git-p4.py#L111

OK, you're right (probably less than ideal behaviour from read_pipe() 
and die() but let's not try to fix that).

quoted

quoted

+                if p4_version_string().find('/NT') >= 0:
+                    text = text.replace('\r\n', '\n')
+                contents = [ text ]

The indentation on this bit doesn't look right to me.

I believe it is exactly how it was:
https://github.com/git/git/blob/97d7ad75b6fe74960d2a12e4a9151a55a5a87d6d/git-p4.py#L2397-L2399

OK.

In general, what is the appropriate way to reference code in this email list? Are GitHub links OK?

I'm not an expert, but it feels possibly a bit ephemeral - if someone is 
digging through email archives in a future where that github project has 
been moved elsewhere, the links will all be dead.

Luke

On 16 Sep 2015, at 00:12, Luke Diamand [off-list ref] wrote:

On 15/09/15 16:38, Lars Schneider wrote:

quoted

On 15 Sep 2015, at 08:43, Luke Diamand [off-list ref] wrote:

quoted

quoted

Do we know the mechanism by which we end up in this state?

Unfortunately no. I tried hard to reproduce the error with “conventional” methods. As you can see I ended up manipulating the P4 database…

However, it looks like this error happens in the wild, too:
https://stackoverflow.com/questions/5156909/translation-of-file-content-failed-error-in-perforce
https://stackoverflow.com/questions/887006/perforce-translation-of-file-content-failed-error

It's described in the Perforce FAQ here:

http://answers.perforce.com/articles/KB/3117

i.e. it looks to be caused by mixing old and new P4 clients.

Good find! No idea why I did not find this article before… I will copy the text from the KB into the git commit message to explain the problem.
 

quoted

quoted

quoted

Known issue: This works only if git-p4 is executed in verbose mode.
In normal mode no exceptions are thrown and git-p4 just exits.

Does that mean that the error will only be detected in verbose mode? That doesn't seem right!

Correct. I don’t like this either but I also don’t want to make huge changes to git-p4.
You can see the root problem here:
https://github.com/git/git/blob/97d7ad75b6fe74960d2a12e4a9151a55a5a87d6d/git-p4.py#L110-L114

Any idea how to approach that best?

I guess what we have is not ideal but probably good enough.

ok. thanks!

I will add another test case without “—verbose" to document that there is work to do :-)

quoted

quoted

quoted

+            try:
+                text = p4_read_pipe(['print', '-q', '-o', '-', '%s@%s' % (file['depotFile'], file['change'])])
+            except Exception as e:

Would it be better to specify which kind of Exception you are catching? Looks like you could get OSError, ValueError and CalledProcessError; it's the last of these you want (I think).

I think it is just a plain exception. See here:
https://github.com/git/git/blob/97d7ad75b6fe74960d2a12e4a9151a55a5a87d6d/git-p4.py#L111

OK, you're right (probably less than ideal behaviour from read_pipe() and die() but let's not try to fix that).

ok

quoted

quoted

quoted

+                if p4_version_string().find('/NT') >= 0:
+                    text = text.replace('\r\n', '\n')
+                contents = [ text ]

The indentation on this bit doesn't look right to me.

I believe it is exactly how it was:
https://github.com/git/git/blob/97d7ad75b6fe74960d2a12e4a9151a55a5a87d6d/git-p4.py#L2397-L2399

OK.

quoted

In general, what is the appropriate way to reference code in this email list? Are GitHub links OK?

I'm not an expert, but it feels possibly a bit ephemeral - if someone is digging through email archives in a future where that github project has been moved elsewhere, the links will all be dead.

Right. However, you could disassemble the URL and use the commit hash, the filename and the line number. They are not ephemeral because they are part of the repo.

Thanks,
Lars