Re: kernel.org mirroring (Re: [GIT PULL] MMC update)

Jakub Narebski wrote:

Sending Last-Modified: should be easy; sending ETag needs some consensus
on the contents: mainly about validation. Responding to If-Modified-Since:
and If-None-Match: should cut at least _some_ of the page generating time.

Definitely.

As I said, I'm not talking (at least now) about saving generated HTML
output. This I think is better solved in caching engine like Squid can
be. Although even here some git specific can be of help: we can invalidate
cache on push, and we know that some results doesn't ever change (well,
with exception of changing output of gitweb).

It depends on how creatively you think ;-)

Consider generating static HTML files on each push, via a hook, for many 
of the toplevel files.  The static HTML would then link to the CGI for 
further dynamic querying of the git database.

What can be _easily_ done:
 * Use post 1.4.4 gitweb, which uses git-for-each-ref to generate summary
   page; this leads to around 3 times faster summary page.

This re-opens the question mentioned earlier, is Kay (or anyone?) still 
actively maintaining gitweb on k.org?

 * Perhaps using projects list file (which can be now generated by gitweb)
   instead of scanning directories and stat()-ing for owner would help
   with time to generate projects lis page

This could be statically generated by a robot.  I think everybody would 
shrink in horror if a human needed to maintain such a file.

What can be quite easy incorporated into gitweb:
 * For immutable pages set Expires: or Cache-Control: max-age (or both)
   to infinity

nice!

 * Generate Last-Modified: for those views where it can be calculated,
   and respond with 304 Not Modified as soon as it can.

agreed

What can be easily done using caching engine:
 * Select top 10 of common queries, and cache them, invalidating cache on push
   (depending on query: for example invalidate project list on push to any
   project, invalidate RSS/Atom feed and summary pages only on push to specific
   project) - can be done with git hooks.

Or simply generate regular filesystem files into the webspace, as 
triggered by a hook.  Let the standard filesystem mirroring/caching work 
its magic.

	Jeff