Re: [PATCH v14 00/21] index-helper/watchman

Just thinking out loud. I've been thinking about this more about this.
After the move from signal-based to unix socket for communication, we
probably are better off with a simpler design than the shm-alike one
we have now.

What if we send everything over a socket or a pipe? Sending 500MB over
a unix socket takes 253ms, that's insignificant when operations on an
index that size usually take seconds. If we send everything over
socket/pipe, we can trust data integrity and don't have to verify,
even the trailing SHA-1 in shm file.

So, what I have in mind is this, at read index time, instead of open a
socket, we run a separate program and communicate via pipes. We can
exchange capabilities if needed, then the program sends the entire
current index, the list of updated files back (and/or the list of dirs
to invalidate). The design looks very much like a smudge/clean filter.

For people who don't want extra daemon, they can write a short script
that saves indexes somewhere in tmpfs, and talk to watchman or
something else. I haven't written this script, but I don't think it
takes long to write one. Windows folks have total freedom to implement
a daemon, a service or whatever and use this program as front end. How
the service talks to this program is totally up to them. For people
who want to centralize everything, they can have just one daemon and
have the script to talk to this daemon.

I can see that getting rid of file-based stuff simplifies some
patches. We can still provide a daemon to do more advanced stuff (or
to make it work out of the box). But it's not a hard requirement and
we probably don't need to include one right now. And I think it makes
it easier to test as well because we can just go with some fake file
monitor service instead of real watchman.
--
Duy

On Sun, Jul 3, 2016 at 9:57 AM, David Turner [off-list ref] wrote:

This addresses comments on v13:
removed unnecessary no_mmap ifdef
add an ifdef in unix-socket
OS X fix for select()
test improvement

Thanks to all for suggestions.

David Turner (10):
  pkt-line: add gentle version of packet_write
  index-helper: log warnings
  unpack-trees: preserve index extensions
  watchman: add a config option to enable the extension
  index-helper: kill mode
  index-helper: don't run if already running
  index-helper: autorun mode
  index-helper: optionally automatically run
  index-helper: indexhelper.exitAfter config
  mailmap: use main email address for dturner

Nguyễn Thái Ngọc Duy (11):
  read-cache: allow to keep mmap'd memory after reading
  unix-socket.c: add stub implementation when unix sockets are not
    supported
  index-helper: new daemon for caching index and related stuff
  index-helper: add --strict
  daemonize(): set a flag before exiting the main process
  index-helper: add --detach
  read-cache: add watchman 'WAMA' extension
  watchman: support watchman to reduce index refresh cost
  index-helper: use watchman to avoid refreshing index with lstat()
  update-index: enable/disable watchman support
  trace: measure where the time is spent in the index-heavy operations

 .gitignore                               |   2 +
 .mailmap                                 |   1 +
 Documentation/config.txt                 |  12 +
 Documentation/git-index-helper.txt       |  86 +++++
 Documentation/git-update-index.txt       |   6 +
 Documentation/technical/index-format.txt |  22 ++
 Makefile                                 |  22 ++
 builtin/gc.c                             |   2 +-
 builtin/update-index.c                   |  15 +
 cache.h                                  |  25 +-
 command-list.txt                         |   1 +
 config.c                                 |   5 +
 configure.ac                             |   8 +
 contrib/completion/git-completion.bash   |   1 +
 daemon.c                                 |   2 +-
 diff-lib.c                               |   4 +
 dir.c                                    |  25 +-
 dir.h                                    |   6 +
 environment.c                            |   2 +
 git-compat-util.h                        |   1 +
 index-helper.c                           | 469 +++++++++++++++++++++++++++
 name-hash.c                              |   2 +
 pkt-line.c                               |  18 ++
 pkt-line.h                               |   2 +
 preload-index.c                          |   2 +
 read-cache.c                             | 531 ++++++++++++++++++++++++++++++-
 refs/files-backend.c                     |   2 +
 setup.c                                  |   4 +-
 t/t1701-watchman-extension.sh            |  37 +++
 t/t7063-status-untracked-cache.sh        |  22 ++
 t/t7900-index-helper.sh                  |  79 +++++
 t/test-lib-functions.sh                  |   4 +
 test-dump-watchman.c                     |  16 +
 unix-socket.h                            |  18 ++
 unpack-trees.c                           |   1 +
 watchman-support.c                       | 135 ++++++++
 watchman-support.h                       |   7 +
 37 files changed, 1578 insertions(+), 19 deletions(-)
 create mode 100644 Documentation/git-index-helper.txt
 create mode 100644 index-helper.c
 create mode 100755 t/t1701-watchman-extension.sh
 create mode 100755 t/t7900-index-helper.sh
 create mode 100644 test-dump-watchman.c
 create mode 100644 watchman-support.c
 create mode 100644 watchman-support.h

--
1.9.1

-- 
Duy