[PATCH] gitweb: speed up project listing by limiting find depth

Subsystems: kernel build + files below scripts/ (unless maintained elsewhere), the rest

DORMANTno replies

3 messages, 2 authors, 2016-06-15 · open the first message on its own page

[PATCH] gitweb: speed up project listing by limiting find depth

From: Luke Lu <hidden>
Date: 2016-06-15 22:43:42

Resubmit patch due to tab/space issue :)

Signed-off-by: Luke Lu <redacted>
---
 Makefile           |    2 ++
 gitweb/gitweb.perl |   11 +++++++++++
 2 files changed, 13 insertions(+), 0 deletions(-)
diff --git a/Makefile b/Makefile
index 8db4dbe..3e9938e 100644
--- a/Makefile
+++ b/Makefile
@@ -165,6 +165,7 @@ GITWEB_CONFIG = gitweb_config.perl
 GITWEB_HOME_LINK_STR = projects
 GITWEB_SITENAME =
 GITWEB_PROJECTROOT = /pub/git
+GITWEB_PROJECT_MAXDEPTH = 2007
 GITWEB_EXPORT_OK =
 GITWEB_STRICT_EXPORT =
 GITWEB_BASE_URL =
@@ -831,6 +832,7 @@ gitweb/gitweb.cgi: gitweb/gitweb.perl
 	    -e 's|++GITWEB_HOME_LINK_STR++|$(GITWEB_HOME_LINK_STR)|g' \
 	    -e 's|++GITWEB_SITENAME++|$(GITWEB_SITENAME)|g' \
 	    -e 's|++GITWEB_PROJECTROOT++|$(GITWEB_PROJECTROOT)|g' \
+	    -e 's|"++GITWEB_PROJECT_MAXDEPTH++"|$(GITWEB_PROJECT_MAXDEPTH)|g' \
 	    -e 's|++GITWEB_EXPORT_OK++|$(GITWEB_EXPORT_OK)|g' \
 	    -e 's|++GITWEB_STRICT_EXPORT++|$(GITWEB_STRICT_EXPORT)|g' \
 	    -e 's|++GITWEB_BASE_URL++|$(GITWEB_BASE_URL)|g' \
diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl
index 3064298..d62357f 100755
--- a/gitweb/gitweb.perl
+++ b/gitweb/gitweb.perl
@@ -35,6 +35,10 @@ our $GIT = "++GIT_BINDIR++/git";
 #our $projectroot = "/pub/scm";
 our $projectroot = "++GITWEB_PROJECTROOT++";
 
+# fs traversing limit for getting project list
+# the number is relative to the projectroot
+our $project_maxdepth = "++GITWEB_PROJECT_MAXDEPTH++";
+
 # target of the home link on top of all pages
 our $home_link = $my_uri || "/";
 
@@ -1509,16 +1513,23 @@ sub git_get_projects_list {
 		# remove the trailing "/"
 		$dir =~ s!/+$!!;
 		my $pfxlen = length("$dir");
+		my $pfxdepth = ($dir =~ tr!/!!);
 
 		File::Find::find({
 			follow_fast => 1, # follow symbolic links
 			follow_skip => 2, # ignore duplicates
+			no_chdir => 1, # don't chdir into every directory
 			dangling_symlinks => 0, # ignore dangling symlinks, silently
 			wanted => sub {
 				# skip project-list toplevel, if we get it.
 				return if (m!^[/.]$!);
 				# only directories can be git repositories
 				return unless (-d $_);
+				# don't traverse too deep (Find is super slow on os x)
+				if (tr!/!! - $pfxdepth > $project_maxdepth) {
+					$File::Find::prune = 1;
+					return;
+				}
 
 				my $subdir = substr($File::Find::name, $pfxlen + 1);
 				# we check related file in $projectroot
-- 
1.5.3.4

Re: [PATCH] gitweb: speed up project listing by limiting find depth

From: Shawn O. Pearce <hidden>
Date: 2016-06-15 22:43:42

Luke Lu [off-list ref] wrote:
Resubmit patch due to tab/space issue :)
Thanks, I have this locally from your prior version but already
had fixed the tab/space problem.
+GITWEB_PROJECT_MAXDEPTH = 2007
Cute.  But does what I was asking for, which was to not change
behavior for existing users.  Most folks have a MAX_PATH around
1024-4096.  There's no sane way they would exceed 2000 nested
directories.

-- 
Shawn.

Re: [PATCH] gitweb: speed up project listing by limiting find depth

From: Shawn O. Pearce <hidden>
Date: 2016-06-15 22:43:42

Luke Lu [off-list ref] wrote:
quoted hunk
diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl
index 3064298..d62357f 100755
--- a/gitweb/gitweb.perl
+++ b/gitweb/gitweb.perl
@@ -1509,16 +1513,23 @@ sub git_get_projects_list {
...
+				# don't traverse too deep (Find is super slow on os x)
+				if (tr!/!! - $pfxdepth > $project_maxdepth) {
+					$File::Find::prune = 1;
+					return;
+				}
 
 				my $subdir = substr($File::Find::name, $pfxlen + 1);
Your patch appears to be causing some errors in the test suite
in t/t9500-gitweb-standalone-no-errors.sh.  Perl is whining about
$subdir not getting initialized above due to the substr being off
the string.  I've got too many other topics tonight to figure out
why yours is failing, can you please run the test and resubmit when
you've resolved the error?

-- 
Shawn.
Keyboard shortcuts
hback out one level
jnext message in thread
kprevious message in thread
ldrill in
Escclose help / fold thread tree
?toggle this help