Re: [RFC PATCH v7 11/9] [PoC] gitweb/lib - tee, i.e. print and capture during cache entry generation

Jakub Narebski wrote:

On Tue, 4 Jan 2011, J.H. wrote:

quoted

On 01/03/2011 01:33 PM, Jakub Narebski wrote:

quoted

quoted

Instead of having gitweb use progress info indicator / throbber to
notify user that data is being generated by current process, gitweb
can now (provided that PerlIO::tee from PerlIO::Util is available)
send page to web browser while simultaneously saving it to cache
(print and capture, i.e. tee), thus having incremental generating of
page serve as a progress indicator.

In general, and particularly for the large sites that caching is
targeted at, teeing is a really bad idea.

[...]

quoted

1) Errors may still be generated in flight as the cache is being
generated.  It would be better to let the cache run with a progress
indicator and should an error occur, display the error instead of giving
any output that may have been generated (and thus likely a broken page).

On the contrary, with tee-ing (and zero size sanity check) you would be
able to see pages even if there are errors saving cache entry.  Though
this wouldn't help very large sites which cannot function without caching,
it could be useful for smaller sites.

I was not sure how Perl reacts to ENOSPC (No space left on device), 
which I think it is only error that can be generated in flight as
cache is being generated (or as gitweb output is printed i.e. sent
to browser and captured/tee-ed i.e. saved to cache entry file), so
I have checked this (using loopback to create small filesystem).

The outcomes one worry about are the following:
* Perl dies during printing - this leads to broken page send to
  browser, and no cache entry generated
* Perl prints output without dying at all; the page send to browser
  via tee-in is all right, but cache entry is truncated which results
  in broken page shown to other clients.

But what actually happens is actually different, and quite safe:
* Perl prints output without dying, and dies on closing cache entry
  file with ENOSPC.  This means that client generating data gets correct
  output, and cache entry is not generated.  Other clients with my code
  try their hand at generation and also get correct page, but not save
  it to cache.

This means that no error page about problems with cache is shown, which
is bad.  On the other hand, at least for smaller sites, gitweb keeps 
working as if without cache for newer entries.
  
Note that observed behaviour might depend on operating system / filesystem
parameters, such as buffer sizes.

But see below.

[...]

Note also that in my rewrite you can simply (by changing one single 
configuration knob) configure gitweb to also cache error pages.  This
might be best and safest solution for very large sites with very large
disk space, but not so good for smaller sites.

Errr... now after rereading your email I see that caching error pages
has one problem: errors that come from the caching engine or capturing
engine - those errors you cannot cache.  Sorry, my mistake.

quoted

- John 'Warthog9' Hawley

P.S. I'm back to work full-time on Wednesday, which I'll be catching up
on gitweb and trying to make forward progress on my gitweb code again.

I'll try to send much simplified (and easier to use in caching) error
handling using exceptions (die / eval used as throw / catch) today.

Sent as
  [RFC PATCH v7 2.5/9] gitweb: Make die_error just die, and use send_error
    to create error pages
  Message-ID: [ref]
  http://permalink.gmane.org/gmane.comp.version-control.git/164466

-- 
Jakub Narebski
Poland