for i in ??
do
cp -r $i $i.new
rm -rf $i
mv $i.new $i
done
in your .git/objects/ directory (CAREFUL! Any script that does "rm -rf"
should be double- and triple-checked for sanity! ;)
Insanity is copying the data rather than just the file name. Git is
good about not reading unnecessary files, and anything necessary should
be cached, so on-disk fragmentation is not a concern.
rmdir --ignore-fail-on-non-empty ?? # Probably unnecessary.
for i in ??
do
mkdir $i.new
mv $i/* $i.new
rmdir $i
mv $i.new $i
done
From: Jon Smirl <hidden> Date: 2016-06-15 22:42:29
On 12 Jun 2006 00:39:49 -0400, linux@horizon.com [off-list ref] wrote:
Insanity is copying the data rather than just the file name. Git is
good about not reading unnecessary files, and anything necessary should
be cached, so on-disk fragmentation is not a concern.
I've run a pack and I moved the objects to new directories. Directory
is 746M with 64K files now.
I've stablized like this. 1GB RAM with 2.8Ghz P4 hyperthread. Is there
anyway to tell what it is doing in the kernel for so much time?
procs -----------memory---------- ---swap-- -----io---- --system--
-----cpu------
r b swpd free buff cache si so bi bo in cs us
sy id wa st
1 0 599128 24712 38196 247008 0 0 0 0 451 382 12
39 48 0 0
1 0 599128 24836 38196 246872 0 0 0 4 413 354 15
38 48 0 0
1 0 599128 24960 38212 246856 0 0 0 64 453 390 15
37 48 0 0
1 0 599128 24960 38212 246856 0 0 0 0 414 367 12
40 49 0 0
1 0 599128 23504 38212 248216 0 0 0 0 448 365 13
39 48 0 0
1 0 599128 24156 38212 247604 0 0 0 0 407 355 13
39 49 0 0
1 0 599128 25240 38212 246652 0 0 0 0 446 390 13
39 48 0 0
1 0 599128 25240 38224 246572 0 0 4 48 415 418 12
40 47 0 0
1 0 599128 25116 38232 246496 0 0 0 12 452 432 12
40 48 0 0
Still doesn't seem to be making much forward progress.
--
Jon Smirl
jonsmirl@gmail.com
I've stablized like this. 1GB RAM with 2.8Ghz P4 hyperthread. Is there
anyway to tell what it is doing in the kernel for so much time?
oprofile will tell you.
I don't see why it would spend a lot of time in the kernel, unless it's
the SVN part that does a ton of reads or something. git should have almost
no kernel footprint apart from the individual objects creation/reading, so
once it's repacked, I generally see very little system time.
What does top say? (Ie can you see _which_ process spends time in the
kernel?)
Linus
From: Jon Smirl <hidden> Date: 2016-06-15 22:42:29
On 6/12/06, Linus Torvalds [off-list ref] wrote:
On Mon, 12 Jun 2006, Jon Smirl wrote:
quoted
I've stablized like this. 1GB RAM with 2.8Ghz P4 hyperthread. Is there
anyway to tell what it is doing in the kernel for so much time?
oprofile will tell you.
I don't see why it would spend a lot of time in the kernel, unless it's
the SVN part that does a ton of reads or something. git should have almost
no kernel footprint apart from the individual objects creation/reading, so
once it's repacked, I generally see very little system time.
What does top say? (Ie can you see _which_ process spends time in the
kernel?)
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
14525 jonsmirl 16 0 604m 391m 1904 S 24 38.7 916:53.39 git-svnimport
20947 jonsmirl 17 0 0 0 0 R 1 0.0 0:00.03 git-svnimport
Hard to tell, it's obviously got short-lived processes there too that it's
not showing, but equally obviously that svnimport script itself is
spending an alarming amount of CPU time. I don't think it should do that
much processing, but since it's written in perl, I can't read it.
Are there any other directories that seem to be growing (eg some temp-file
directory where the old files aren't cleaned away?). I can't imagine what
else it could be doing in kernel space than simply some silly filesystem
operation, but dang it all, Linux filesystems are usually very efficient
indeed, unless we're talking huge directories (and if it's not the git
object directory any more, it must be something else).
At least with the cvs importer I have _some_ clue what it's doing, since I
wrote an earlier version myself (very different, but at least I know what
the operations are). SVN has always just confused me, and I have no idea
what svnimport does, so I think I'll have to defer to somebody who
actually knows the code.
Smurf, have you looked at any larger repositories?
Linus
From: Jon Smirl <hidden> Date: 2016-06-15 22:42:29
On 6/12/06, Linus Torvalds [off-list ref] wrote:
On Mon, 12 Jun 2006, Jon Smirl wrote:
quoted
I've stablized like this. 1GB RAM with 2.8Ghz P4 hyperthread. Is there
anyway to tell what it is doing in the kernel for so much time?
oprofile will tell you.
I don't have profiling turn on in the kernel. I've turned it on so
I'll pick it up next time I reboot.
I'll kill everything and restart when my new RAM arrives tomorrow.
Hopefully the SVN import will finish before then but it doesn't look likely.
--
Jon Smirl
jonsmirl@gmail.com
From: Jon Smirl <hidden> Date: 2016-06-15 22:42:29
On 6/12/06, Linus Torvalds [off-list ref] wrote:
On Mon, 12 Jun 2006, Jon Smirl wrote:
quoted
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
14525 jonsmirl 16 0 604m 391m 1904 S 24 38.7 916:53.39 git-svnimport
20947 jonsmirl 17 0 0 0 0 R 1 0.0 0:00.03 git-svnimport
Hard to tell, it's obviously got short-lived processes there too that it's
not showing, but equally obviously that svnimport script itself is
spending an alarming amount of CPU time. I don't think it should do that
much processing, but since it's written in perl, I can't read it.
Are there any other directories that seem to be growing (eg some temp-file
directory where the old files aren't cleaned away?). I can't imagine what
else it could be doing in kernel space than simply some silly filesystem
operation, but dang it all, Linux filesystems are usually very efficient
indeed, unless we're talking huge directories (and if it's not the git
object directory any more, it must be something else).
64 files in tmp.
But the SVN repository itself has 411,000 files in it. Split between
two directories.
Is there some pack equivalent for svn that I haven't found yet?
At least with the cvs importer I have _some_ clue what it's doing, since I
wrote an earlier version myself (very different, but at least I know what
the operations are). SVN has always just confused me, and I have no idea
what svnimport does, so I think I'll have to defer to somebody who
actually knows the code.
Smurf, have you looked at any larger repositories?
Linus
From: Jon Smirl <hidden> Date: 2016-06-15 22:42:29
On 6/12/06, Jon Smirl [off-list ref] wrote:
On 6/12/06, Linus Torvalds [off-list ref] wrote:
quoted
On Mon, 12 Jun 2006, Jon Smirl wrote:
quoted
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
14525 jonsmirl 16 0 604m 391m 1904 S 24 38.7 916:53.39 git-svnimport
20947 jonsmirl 17 0 0 0 0 R 1 0.0 0:00.03 git-svnimport
Hard to tell, it's obviously got short-lived processes there too that it's
not showing, but equally obviously that svnimport script itself is
spending an alarming amount of CPU time. I don't think it should do that
much processing, but since it's written in perl, I can't read it.
Are there any other directories that seem to be growing (eg some temp-file
directory where the old files aren't cleaned away?). I can't imagine what
else it could be doing in kernel space than simply some silly filesystem
operation, but dang it all, Linux filesystems are usually very efficient
indeed, unless we're talking huge directories (and if it's not the git
object directory any more, it must be something else).
64 files in tmp.
But the SVN repository itself has 411,000 files in it. Split between
two directories.
I'm doing all of this on ext3. I have plenty of free disk space so I
can make another partition and switch to a new file system after I
install the new RAM. What would be the best one to try? Doing that
would provide a data point to determine if this is a problem with file
system performance or the misuse of file systems.
--
Jon Smirl
jonsmirl@gmail.com
64 files in tmp.
But the SVN repository itself has 411,000 files in it. Split between
two directories.
Ouch. That sounds like it.
Is there some pack equivalent for svn that I haven't found yet?
Is this literally what SVN does normally? That's just insane. I mean, even
git tried to at least hash out the files (and yeah, admittedly even that
worked less well than I was hoping for, but I at least fixed it within
just a few weeks through the pack mechanism).
Or is that 411,000 files a result of how git-svnimport does things, rather
than some basic SVN approach to live: does it perhaps end up checking out
each file under an individual temporary name?
Linus
From: Jon Smirl <hidden> Date: 2016-06-15 22:42:29
On 6/12/06, Linus Torvalds [off-list ref] wrote:
quoted
Is there some pack equivalent for svn that I haven't found yet?
Is this literally what SVN does normally? That's just insane. I mean, even
git tried to at least hash out the files (and yeah, admittedly even that
worked less well than I was hoping for, but I at least fixed it within
just a few weeks through the pack mechanism).
Or is that 411,000 files a result of how git-svnimport does things, rather
than some basic SVN approach to live: does it perhaps end up checking out
each file under an individual temporary name?
The svn repository was built by cvs2svn, none of the git tools were involved.
--
Jon Smirl
jonsmirl@gmail.com
64 files in tmp.
But the SVN repository itself has 411,000 files in it. Split between
two directories.
I'm doing all of this on ext3. I have plenty of free disk space so I
can make another partition and switch to a new file system after I
install the new RAM. What would be the best one to try? Doing that
would provide a data point to determine if this is a problem with file
system performance or the misuse of file systems.
I'm sure there are better filesystems to try for this kind of insane
schenario, but at the same time, I really cannot imaging that the 411,000
files is a "normal" thing. There _must_ be some way to have SVN not do
that in the first place (or git-svnimport).
Is this what happened when the SVN people started using fsfs?
Linus
The svn repository was built by cvs2svn, none of the git tools were involved.
Ok, so that part is purely a SVN issue.
Having that many files in a single directory (or two) is a total disaster.
That said, it works well enough if you don't create new files very often
(and _preferably_ don't look them up either, although that is effectively
helped by indexing). I _suspect_ that
- the "cvs->svn" import process was probably optimized so that it did one
file at a time (your "eight stages" description certainly sounds as if
it could do it), and in that case it's entirely possible that that can
be done efficiently (ie you still do file creates and lookups in an
increasingly big directory, but you do it only _once_ per file, rather
than look up old files all the time). So your lookup ratio would be 1:1
with the files.
Doing a git-cvsimport would then do basically random lookups in that
_huge_ directory, and instead of reading the files one at a time (and
fully) and never again, I assume it opens them, reads one revision,
closes it, and then goes on to the next revision, so it will have a
much higher lookup ratio (you'd look up every file several times).
- I suspect the SVN people must be hurting for performance themselves. I
guess they don't expect to be able to do 5-10 commits per second, the
way git was designed to do. So they optimized the cvs import part, but
their actual regular live usage is probably hitting this same directory
inefficiency.
Of course, the old SVN Berkeley DB usage was probably even worse (not in
system time, but I'd expect the access patterns within the BDB file to be
pretty nasty, and probably a lot of user time spent seeking around it).
But in this particular case, it might even have been better.
Maybe we could teach the SVN people about pack-files? ;)
Linus
From: Jon Smirl <hidden> Date: 2016-06-15 22:42:29
On 6/12/06, Linus Torvalds [off-list ref] wrote:
Having that many files in a single directory (or two) is a total disaster.
That said, it works well enough if you don't create new files very often
(and _preferably_ don't look them up either, although that is effectively
helped by indexing). I _suspect_ that
Posted to the svn list, they said that 220K files is normal. They told
me to turn on the ext2 dir_index option. Cheking my system I see that
none of partitions have it turned on so it must not be the default for
FC5.
I have to unmount the drive to convert existing directories. I can
trying doing the file move trick while the process is running since
new directories will use it.
--
Jon Smirl
jonsmirl@gmail.com
From: Jon Smirl <hidden> Date: 2016-06-15 22:42:29
On 6/12/06, Jon Smirl [off-list ref] wrote:
On 6/12/06, Linus Torvalds [off-list ref] wrote:
quoted
Having that many files in a single directory (or two) is a total disaster.
That said, it works well enough if you don't create new files very often
(and _preferably_ don't look them up either, although that is effectively
helped by indexing). I _suspect_ that
Posted to the svn list, they said that 220K files is normal. They told
me to turn on the ext2 dir_index option. Cheking my system I see that
none of partitions have it turned on so it must not be the default for
FC5.
I have to unmount the drive to convert existing directories. I can
trying doing the file move trick while the process is running since
new directories will use it.
I converted the ext3 directories to dir_index on the fly using the
move trick. Switching the directory index makes it look like it is
spending even more time in the kernel.
procs -----------memory---------- ---swap-- -----io---- --system--
-----cpu------
r b swpd free buff cache si so bi bo in cs us
sy id wa st
1 0 636188 22380 19176 157200 0 0 0 52 436 415 13
40 48 0 0
1 0 636188 22504 19176 157200 0 0 0 0 430 373 13
38 49 0 0
1 0 636188 22628 19176 157064 0 0 0 0 433 380 12
39 49 0 0
1 0 636188 22628 19184 157056 0 0 0 20 434 390 12
38 49 0 0
1 0 636188 22628 19184 156920 0 0 0 0 431 376 11
40 49 0 0
1 0 636188 22752 19192 156912 0 0 0 48 437 376 12
40 49 0 0
1 0 636188 22876 19192 156912 0 0 0 0 430 386 11
40 49 0 0
1 0 636188 22752 19192 156776 0 0 0 0 431 370 10
41 49 0 0
1 0 636188 23016 19192 156776 0 0 8 0 422 500 22
40 37 2 0
The size of the svn directories went from 3.2MB to 4.4MB after they
were converted to ext3 indexed mode.
I'll get oprofile running when I do a reboot.
--
Jon Smirl
jonsmirl@gmail.com