Re: Issue: repack semi-frequently fails on Windows (msysgit) - suspecting file descriptor issues

Hi,

sorry, I had sent the prior mail prematurely (hit wrong key)
and have been busy working on the resubmission...

On Thu, Apr 16, 2015 at 01:10:36PM +0200, Thomas Braun wrote:

Am 16.04.2015 um 12:03 schrieb Andreas Mohr:

quoted

Hi all,

over the years I've had the same phenomenon with various versions of msysgit
(now at 1.9.5.msysgit.0, on Windows 7 64bit), so I'm now sufficiently
confident of it being a long-standing, longer-term issue and thus I'm
reporting it now.

(I've had experience with this issue as early as 1.7.x, I believe).

(CC'ing msysgit)

Hi Andreas,

quoted

Since I'm doing development in a sufficiently rebase-heavy manner,
I seem to aggregate a lot of objects.
Thus, when fetching content I'm sufficiently frequently greeted with
a git gc run.
This, however, does not work fully reliably:

    Auto packing the repository for optimum performance. You may also
    run "git gc" manually. See "git help gc" for more information.
    Counting objects: 206527, done.
    Delta compression using up to 4 threads.
    Compressing objects: 100% (27430/27430), done.
    Writing objects: 100% (206527/206527), done.
    Total 206527 (delta 178632), reused 206527 (delta 178632)
    Unlink of file '.git/objects/pack/pack-ab1712db0a94b5c55538d3b4cb3660cedc264c3c.pack' failed. Should I try again? (y/n) n
    Unlink of file '.git/objects/pack/pack-ab1712db0a94b5c55538d3b4cb3660cedc264c3c.idx' failed. Should I try again? (y/n) n
    Checking connectivity: 206527, done.

A workable workaround for this recurring issue
(such a fetch will fail repeatedly,
thereby hampering my ability to update properly)
is to manually do a "git gc --auto"
prior to the fetch (which will then succeed).

I've never had this issue. The error message from unlinking the file
means that someone is still accessing the file and thus it can not be
deleted (due to the implicit file locking on windows).

Can you reproduce the error reliably?

It seems to be reproducible pretty reliably,
at least once git thinks it needs to repack (initiated by a fetch operation, I think),
*and* then the unlink issue successfully turned up
(which may happen perhaps every 20 fetches of a *very* rebase-heavy workflow).


Interim mail content:

I strongly suspect that git's repacking implementation
(probably unrelated to msysgit-specific deviations,
IOW, git *core* handling)
simply is buggy
in that it may keep certain file descriptors open
at least a certain time (depending on scope of implementation/objects!?)
beyond having finished its operation (rename?).
As a related note, in an unrelated application of mine
I also encountered issues on Windows
where renaming of in-use files and further use of these files/names
then failed (error code was EACCES I believe).
IOW, this seems to be an issue specific to
Windows' "special" (and sometimes quirky) filesystem handling
which probably does not turn up on many "other" platforms,
thus a historic existing implementation weakness in git's repack handling
could not be nailed down in a sufficiently easy manner.




I think I may have the order wrong, however:
Handling seems to be:
- repack needed
- counting objects
- compressing
- writing
- unlink (delete) of all prior non-repacked objects (which fails)


I have to admit that at this point in time I'm actually unsure
which higher-level operation it actually is
that gets carried out where eventually a repack *implicitly* gets triggered
(I've got a shell script here which implements clean branch updating,
where I eventually hit the problem during its daily use).


Since a standalone git gc --auto *immediately* appears to work
(after many repeated attempts of failing full-update),
this is a strong hint that (in the failure case)
it's the *PRIOR* (non-repack) operation
which has kept these objects open beyond its actual operation scope.


Suspected implementation sample code:

if (operation_needed)
{
  operation_workingset set;

  set.DoStuff();

  if (repack_needed)
  {
    repack_handler repack;

     repack.DoStuff();
  }
}

[NOTE the very prominent scope issues in this example,
which might be the exact reason for hitting such unlink failures -
simply due to having kept file descriptors open within the working set]

I have not had a look at git source though
to actually determine whether there do exist
such severe operation scope issues
that I'm strongly contemplating.

Andreas Mohr