Re: kernel.org mirroring (Re: [GIT PULL] MMC update)

Linus Torvalds wrote:
[...] 

For example, if the git "refs/heads/" (or tags) directory hasn't changed 
in the last two months, we should probably set any ref-relative gitweb 
pages to have a caching timeout of a day or two. In contrast, if it's 
changed in the last hour, maybe we should only cache it for five minutes.

Jakub: any way to make gitweb set the "expires" fields _much_ more 
aggressively. I think we should at least have the ability to set a basic 
rules like

 - a _minimum_ of five minutes regardless of anything else

   We might even tweak this based on loadaverage, and it might be 
   worthwhile to add a randomization, to make sure that you don't get into 
   situations where everything webpage needs to be recalculated at once.

I think the minimum expires (or minimum _additional_ expires: as of now
giweb only does expires +1d for explicit hash requests) should depend on
how often project changes. How often there are pushes to kernel.org?
 

 - if refs/ directories are old, raise the minimum by the age of the refs

   If it's more than an hour old, raise it to ten minutes. If it's more 
   than a day, raise it to an hour. If it's more than a month old, raise 
   it to a day. And if it's more than half a year, it's some historical 
   archive like linux-history, and should probably default to a week or 
   more.

What about packed refs?

We can certainly raise expires for tags (tags objects), as they should not
usually change.
 

 - infinite for stuff that isn't ref-related.

As sha1 is not changeable, everything that is accessed by explicit 
sha1 (hash), or by explicit sha1 (hash_base) plus pathname (file_name)
should have effectively infinite expires.


Every caching would need some temporary memory, or temporary disk space.
And perhaps mod_perl specific caching would be useful here...

P.S. I have added Pasky to Cc:, as he manages http://repo.or.cz public
git repository hosting (much smaller than kernel.org and I think under less
load: but also I think withour kernel.org resources).
-- 
Jakub Narebski