[PATCH 1/2] Added support for purged files and also optimised memory usage.

Purged files are handled as if they are merely deleted, which is not
entirely optimal, but I don't know of any other way to handle them.
File data is deleted from memory as early as they can, and they are more
efficiently handled, at (significant) cost to CPU usage.

Still need to handle p4 branches with spaces in their names.
Still need to make git-p4 clone more reliable.
 - Perhaps with a --continue option. (Sometimes the p4 server kills
 the connection)

Signed-off-by: John Chapman <redacted>
---
 contrib/fast-import/git-p4 |   14 +++++++-------
 1 files changed, 7 insertions(+), 7 deletions(-)

diff --git a/contrib/fast-import/git-p4 b/contrib/fast-import/git-p4
index 2216cac..38d1a17 100755
--- a/contrib/fast-import/git-p4
+++ b/contrib/fast-import/git-p4

@@ -946,7 +946,7 @@ class P4Sync(Command):
 
             if includeFile:
                 filesForCommit.append(f)
-                if f['action'] != 'delete':
+                if f['action'] not in ('delete', 'purge'):
                     filesToRead.append(f)
 
         filedata = []

@@ -965,11 +965,11 @@ class P4Sync(Command):
         while j < len(filedata):
             stat = filedata[j]
             j += 1
-            text = [];
+            text = ''
             while j < len(filedata) and filedata[j]['code'] in ('text', 'unicode', 'binary'):
-                text.append(filedata[j]['data'])
+                text += filedata[j]['data']
+                del filedata[j]['data']
                 j += 1
-            text = ''.join(text)
 
             if not stat.has_key('depotFile'):
                 sys.stderr.write("p4 print fails with: %s\n" % repr(stat))

@@ -1038,7 +1038,7 @@ class P4Sync(Command):
                 continue
 
             relPath = self.stripRepoPath(file['path'], branchPrefixes)
-            if file["action"] == "delete":
+            if file["action"] in ("delete", "purge"):
                 self.gitStream.write("D %s\n" % relPath)
             else:
                 data = file['data']

@@ -1077,7 +1077,7 @@ class P4Sync(Command):
 
                 cleanedFiles = {}
                 for info in files:
-                    if info["action"] == "delete":
+                    if info["action"] in ("delete", "purge"):
                         continue
                     cleanedFiles[info["depotFile"]] = info["rev"]
 

@@ -1400,7 +1400,7 @@ class P4Sync(Command):
             if change > newestRevision:
                 newestRevision = change
 
-            if info["action"] == "delete":
+            if info["action"] in ("delete", "purge"):
                 # don't increase the file cnt, otherwise details["depotFile123"] will have gaps!
                 #fileCnt = fileCnt + 1
                 continue

-- 
1.6.0.3.643.g233db

Signed-off-by: John Chapman <redacted>
---
 contrib/fast-import/git-p4 |    5 ++++-
 1 files changed, 4 insertions(+), 1 deletions(-)

diff --git a/contrib/fast-import/git-p4 b/contrib/fast-import/git-p4
index 38d1a17..9f0a5f9 100755
--- a/contrib/fast-import/git-p4
+++ b/contrib/fast-import/git-p4

@@ -316,8 +316,11 @@ def gitBranchExists(branch):
                             stderr=subprocess.PIPE, stdout=subprocess.PIPE);
     return proc.wait() == 0;
 
+_gitConfig = {}
 def gitConfig(key):
-    return read_pipe("git config %s" % key, ignore_error=True).strip()
+    if not _gitConfig.has_key(key):
+        _gitConfig[key] = read_pipe("git config %s" % key, ignore_error=True).strip()
+    return _gitConfig[key]
 
 def p4BranchesInGit(branchesAreInRemotes = True):
     branches = {}

-- 
1.6.0.3.643.g233db

On Fri, Nov 7, 2008 at 7:22 PM, John Chapman [off-list ref] wrote:

+_gitConfig = {}
 def gitConfig(key):
-    return read_pipe("git config %s" % key, ignore_error=True).strip()
+    if not _gitConfig.has_key(key):
+        _gitConfig[key] = read_pipe("git config %s" % key, ignore_error=True).strip()
+    return _gitConfig[key]

If this is truly a noticeable bottleneck on Windows, something like
the following might be even better:  (completely untested!)

_gitConfig = None
def gitConfig(key):
  if _gitConfig is None:
    lines = read_pipe("git config -l", ignore_error=True).readlines():
    _gitConfig = dict([l.strip().split('=', 1) for l in lines])
  return _gitConfig.get(key, None)



Dave.

On Fri, 2008-11-07 at 21:19 -0800, David Symonds wrote:
<snip>

_gitConfig = None
def gitConfig(key):
  if _gitConfig is None:
    lines = read_pipe("git config -l", ignore_error=True).readlines():
    _gitConfig = dict([l.strip().split('=', 1) for l in lines])
  return _gitConfig.get(key, None)

That certainly is better, if one can assume that git's configuration is
small. (And relative to the memory usage of the script, it will
definetly be small).

I shall give that a go, although the change won't make it even faster -
I suspect that much of the performance penalty in windows is the
pathetic fork() performance, particularly as the memory usage of the
script increases. (If subprocess does fork() and exec() in order to open
another process, in cygwin).

Thankyou.

"David Symonds" [off-list ref] writes:

On Fri, Nov 7, 2008 at 7:22 PM, John Chapman [off-list ref] wrote:

quoted

+_gitConfig = {}
 def gitConfig(key):
-    return read_pipe("git config %s" % key, ignore_error=True).strip()
+    if not _gitConfig.has_key(key):
+        _gitConfig[key] = read_pipe("git config %s" % key, ignore_error=True).strip()
+    return _gitConfig[key]

If this is truly a noticeable bottleneck on Windows, something like
the following might be even better:  (completely untested!)

_gitConfig = None
def gitConfig(key):
  if _gitConfig is None:
    lines = read_pipe("git config -l", ignore_error=True).readlines():
    _gitConfig = dict([l.strip().split('=', 1) for l in lines])
  return _gitConfig.get(key, None)

Wouldn't it be better to use "git config -l -z", split lines at "\0"
(NUL), and split key from value at first "\N" (CR)? This format was
meant for scripts.

-- 
Jakub Narebski
Poland
ShadeHawk on #git

Arafangion wrote:

On Fri, 2008-11-07 at 21:19 -0800, David Symonds wrote:
<snip>

quoted

_gitConfig = None
def gitConfig(key):
  if _gitConfig is None:
    lines = read_pipe("git config -l", ignore_error=True).readlines():
    _gitConfig = dict([l.strip().split('=', 1) for l in lines])
  return _gitConfig.get(key, None)

That certainly is better, if one can assume that git's configuration is
small. (And relative to the memory usage of the script, it will
definetly be small).

What about using git config --get-regexp to only get the p4-related 
settings ?

I don't really know the options used by git-p4, but something like this 
seems like a good candidate to address every trade-off concerns?

git-config --get-regexp '^(p4|user)\.'