Re: kernel.org mirroring (Re: [GIT PULL] MMC update)

On 12/7/06, Linus Torvalds [off-list ref] wrote:

Does anybody who is web-server clueful know if there is some simple
front-end (squid?) that is easy to set up and can just act as a caching
proxy in front of such an incompetent server?

Squid in "transparent reverse proxy" mode isn't a bad choice, although
I don't know offhand whether it queues/clusters concurrent requests
for the same URL in the way you want.  I suggest the "transparent"
deployment (netfilter/netlink integration) because you can slap it in
with no changes to the origin server and yank it out again if you have
a problem.  The challenge is in getting conntrack to scale to a
zillion concurrent sessions, but you could probably find someone in
your crowd who knows something about that.  :-)

Ignore any documentation that talks about httpd_accel_*.  Configuring
transparent mode is a great deal simpler and saner in squid 2.6 than
it used to be; you just add a "transparent" parameter to the http_port
tag.  With or without this tag, you set up what used to be called
"accelerator mode" using some parameters to http_port and cache_peer,
as described in
http://www.squid-cache.org/mail-archive/squid-users/200607/0162.html.

If transparent mode looks like the right thing for kernel.org, you
might be interested in some netfilter hackery to offload part of the
conntrack session lookup load to a front-end box that blocks DDoS and
acts more or less as an L4 switch plus session context cache.  I've
been banging on a proof of concept implementation for a while, and am
currently working on integrating against 2.6.19 by splitting
nf_conntrack into front and back halves that interact via a sort of
Layer 2+ header.  I have no idea yet whether it will have any
scalability benefit on dual-x86_64 class hardware (it was originally
conceived for rigid cache architectures where the random access
patterns of session lookups have drastic cache effects).

Cheers,