Re: git-reflog 70 minutes at 100% cpu and counting

On Wed, 2009-12-16 at 16:06 -0500, Nicolas Pitre wrote:

On Wed, 16 Dec 2009, Eric Paris wrote:

quoted

On Tue, 2009-12-15 at 22:03 -0500, Nicolas Pitre wrote:

quoted

On Mon, 14 Dec 2009, Eric Paris wrote:

quoted

The alternative repo is slowing pushing up to that same location.  That
tar is 855838982, so just a tad bit smaller.

It doesn't appear to be complete yet, and not progressing either.

The alternative repo is now available (but the original is down)

I tried to run git gc --aggressive last night while I slept and got this
as output, maybe it helps point to a solution/problem?  The git reflog
portion ran for 5 hours and 36 minutes and appears to have finished.

Yes.  I was able to reproduce your issue.  And because of the *horrible* 
repository packing, the reflog expiration process is taking ages when 
determining object reachability at a rate of one reflog entry every 2 
seconds or so.  With 4214 entries for the fsnotify-syscall branch, and 
1352 entries for the fsnotify branch, this already takes up asignificant 
portion of the actual run time.  I'm sure if your repository was 
properly packed this would take less than a minute.

I'm guessing this is a result of stgit.?  These branches really should
be just a branch from a tag (which exists in kernel-1) and about 30-50
patches linearly applied on top.  I don't know how I get that many
objects.  I'm guessing many/most of them are crap that should be able to
be cleaned/deleted entirely as the rebasing/pushing/poping/updating that
stgit does under the covers should have rendered them pointless.  Not
really sure when/how that should/could have happened.

Should I be running git-gc every night?

Now, repacking doesn't work because...

quoted

$ git gc --aggressive
error: Could not read d936ff8a7b0841b51ddf96afa24a30b016824cb2
error: Could not read 29b6c2fb1390b4fd350a5ecc78f1156fc5d91e9f

/me is pretty git dumb, but is there some way to figure out the parents
or children of these?  I just trolled through all of my directories
doing git show and didn't get any hits.  I guess I'll just clean up and
start over....

Those objects are indeed missing from the repository.  Without them your 
repository is "broken".  Either you can find them somewhere else and 
copy them over, or salvage as much as you can by fetching the 
interesting branches into another freshly made repository.  This is 
unfortunate because I would have liked to see by how much this 
repository would have shrunk after a successful repack.

Of course, usage of alternates is recommended _only_ with repositories 
that are stable, i.e. don't ever add repositories to 
.git/objects/info/alternates if those repositories are rewinded/rebased 
and/or branches in them are deleted/replaced.  That could be a reason 
why some objects are now missing from the repository using alternates.

So I'm not sure how I did things wrong.  my kernel-1 has those bunch of
remotes.  The linux-next remote, like I said, basically rebases to
linus' tree, then merges 150 random branches.  It tags that tree every
day and I pull those tags.  So I would never expect any objects from
those remote trees to ever disappear.

Now I created branches in kernel-1 and I certainly have done lots of
things like so

git checkout -b testing remotes/linux-next/master
[edit]
git commit -a
git checkout -b testing1 remotes/linux-next/master
git branch -D testing

My assumption though was that this wouldn't ever affect my other
repositories.  My other repository branches always started by checking
out a branch with remotes/*/* as the base.

My understanding was that I would only run into problems if I used
something on a branch I created myself in the alternatives repo in other
repos (and I didn't remove remotes)

I guess it's not impossible to believe that at some point in time i
would have exported patches to and mbox from kernel-1 and applied them
to kernel-2 or vice versa.  I guess this would end up with the same
objects, right?  Then if I deleted the branch in kernel-1 I would have
problems in kernel-2?

I guess I'll rebuild my setup

new kernel-alt has just the remotes, and my kernel-1,2,3 all alt to it
I'll never have local branches in my kernel-alt
I'll run git-gc every night
I'll hope to never have problem again.

Sound good?