Re: inotify to minimize stat() calls

On Mon, Feb 11, 2013 at 3:16 AM, Junio C Hamano [off-list ref] wrote:

The other "lstat()" experiment was a very interesting one, but this
is not yet an interesting experiment to see where in the "ignore"
codepath we are spending times.

We know that we can tell wt_status_collect_untracked() not to bother
with the untracked or ignored files with !s->show_untracked_files
already, but I think the more interesting question is if we can show
the untracked files with less overhead.

If we want to show untrackedd files, it is a given that we need to
read directories to see what paths there are on the filesystem. Is
the opendir/readdir cost dominating in the process? Are we spending
a lot of time sifting the result of opendir/readdir via the ignore
mechanism? Is reading the "ignore" files costing us much to prime
the ignore mechanism?

If readdir cost is dominant, then that makes "cache gitignore" a
nonsense proposition, I think.  If you really want to "cache"
something, you need to have somebody (i.e. a daemon) who constantly
keeps an eye on the filesystem changes and can respond with the up
to date result directly to fill_directory().  I somehow doubt that
it is a direction we would want to go in, though.

Yeah, it did not cut out syscall cost, I also cut a lot of user-space
processing (plus .gitignore content access). From the timings I posted
earlier,

        unmodified  dir.c
real    0m0.550s    0m0.287s
user    0m0.305s    0m0.201s
sys     0m0.240s    0m0.084s

sys time is reduced from 0.24s to 0.08s, so readdir+opendir definitely
has something to do with it (and perhaps reading .gitignore). But it
also reduces user time from 0.305 to 0.201s. I don't think avoiding
readdir+openddir will bring us this gain. It's probably the cost of
matching .gitignore. I'll try to replace opendir+readdir with a
no-syscall version. At this point "untracked caching" sounds more
feasible (and less complex) than ".gitignore cachine".
-- 
Duy