From: Christian Couder <hidden> Date: 2016-06-15 23:07:01
Hi everyone,
I am starting to investigate ways to speed up git status and other git
commands for Booking.com (thanks to AEvar) and I'd be happy to discuss
the current status or be pointed to relevant documentation or mailing
list threads.
From the threads below ([0], [1], [2], [3], [4], [5], [6], [7], [8]) I
On Thu, Oct 22, 2015 at 7:59 AM, Christian Couder
[off-list ref] wrote:
Hi everyone,
I am starting to investigate ways to speed up git status and other git
commands for Booking.com (thanks to AEvar) and I'd be happy to discuss
the current status or be pointed to relevant documentation or mailing
list threads.
From the threads below ([0], [1], [2], [3], [4], [5], [6], [7], [8]) I
understand that the status is roughly the following:
- instead of working on inotify support it's better to work on using a
cross platform tool like Watchman
Definitely. Especially because watchman has recently gained
(experimental?) Windows support
- instead of working on Watchman support it is better to work first on
caching information in the index
- git update-index --untracked-cache has been developed by Duy and
others and merged to master in May 2015 to cache untracked status in
the index; it is still considered experimental
- git index-helper has been worked on by Duy but its status is not
clear (at least to me)
My roadmap is speeding up index write speed (split index), then read
speed (index-helper) and watchman can be run on top of the
index-helper. Untracked cache solves another performance problem with
.gitignore. All these four pieces are big and getting slowly into
git.git.
The last piece is using watchman with untracked cache to kill the
(small) last stream of lstat() calls. But we'll see if we actually
need it.
What are the possible/planned next steps in this area? improving
--untracked-cache? git index-helper? watchman support?
The index-helper needs some polishing and perhaps more eyeballing. I
have the watchman patch on top of it, but I don't remember if I have
ever sent it out. If anyone wants to help I can resend everything I
have. I don't think I can resume the work soon.
--
Duy
From: David Turner <hidden> Date: 2016-06-15 23:07:04
On Thu, 2015-10-22 at 07:59 +0200, Christian Couder wrote:
Hi everyone,
I am starting to investigate ways to speed up git status and other git
commands for Booking.com (thanks to AEvar) and I'd be happy to discuss
the current status or be pointed to relevant documentation or mailing
list threads.
From the threads below ([0], [1], [2], [3], [4], [5], [6], [7], [8]) I
understand that the status is roughly the following:
- instead of working on inotify support it's better to work on using a
cross platform tool like Watchman
- instead of working on Watchman support it is better to work first on
caching information in the index
- git update-index --untracked-cache has been developed by Duy and
others and merged to master in May 2015 to cache untracked status in
the index; it is still considered experimental
- git index-helper has been worked on by Duy but its status is not
clear (at least to me)
Is that correct?
What are the possible/planned next steps in this area? improving
We're using Watchman at Twitter. A week or two ago posted a dump of our
code to github, but I would advise waiting a day or two to use it, as
I'm about to pull a large number of bugfixes into it (I'll update this
thread and provide a link once I do so).
It's good, but it's not great. One major problem is a bug on OS X[1]
that causes missed updates. Another is that wide changes end up being
quite inefficient when querying watchman. This means that we do some
hackery to manually update the fs_cache during various large git
operations.
I agree that in general it would be better to store or all some of this
information in the index, and the untracked-cache is a good step on
that. But with it enabled and watchman disabled, there still appears to
be 1 lstat per file (plus one stat per dir). The stats per-directory
alone are a large issue for Twitter because we have a relatively deep
and bushy directory structure (an average dir has about 3 or 4 entries
in it). As a result, git status with watchman is almost twice as fast
as with the untracked cache (on my particular machine).
[1] https://github.com/facebook/watchman/issues/172
From: Christian Couder <hidden> Date: 2016-06-15 23:07:06
On Wed, Oct 28, 2015 at 12:54 AM, David Turner [off-list ref] wrote:
On Thu, 2015-10-22 at 07:59 +0200, Christian Couder wrote:
quoted
Hi everyone,
I am starting to investigate ways to speed up git status and other git
commands for Booking.com (thanks to AEvar) and I'd be happy to discuss
the current status or be pointed to relevant documentation or mailing
list threads.
From the threads below ([0], [1], [2], [3], [4], [5], [6], [7], [8]) I
understand that the status is roughly the following:
- instead of working on inotify support it's better to work on using a
cross platform tool like Watchman
- instead of working on Watchman support it is better to work first on
caching information in the index
- git update-index --untracked-cache has been developed by Duy and
others and merged to master in May 2015 to cache untracked status in
the index; it is still considered experimental
- git index-helper has been worked on by Duy but its status is not
clear (at least to me)
Is that correct?
What are the possible/planned next steps in this area? improving
We're using Watchman at Twitter. A week or two ago posted a dump of our
code to github, but I would advise waiting a day or two to use it, as
I'm about to pull a large number of bugfixes into it (I'll update this
thread and provide a link once I do so).
Great, I will have a look at it then!
It's good, but it's not great. One major problem is a bug on OS X[1]
that causes missed updates. Another is that wide changes end up being
quite inefficient when querying watchman. This means that we do some
hackery to manually update the fs_cache during various large git
operations.
I agree that in general it would be better to store or all some of this
information in the index, and the untracked-cache is a good step on
that. But with it enabled and watchman disabled, there still appears to
be 1 lstat per file (plus one stat per dir). The stats per-directory
alone are a large issue for Twitter because we have a relatively deep
and bushy directory structure (an average dir has about 3 or 4 entries
in it). As a result, git status with watchman is almost twice as fast
as with the untracked cache (on my particular machine).
From: David Turner <hidden> Date: 2016-06-15 23:07:09
On Thu, 2015-10-29 at 09:10 +0100, Christian Couder wrote:
quoted
We're using Watchman at Twitter. A week or two ago posted a dump of our
code to github, but I would advise waiting a day or two to use it, as
I'm about to pull a large number of bugfixes into it (I'll update this
thread and provide a link once I do so).
On Mon, Nov 2, 2015 at 9:56 PM, David Turner [off-list ref] wrote:
On Thu, 2015-10-29 at 09:10 +0100, Christian Couder wrote:
quoted
quoted
We're using Watchman at Twitter. A week or two ago posted a dump of our
code to github, but I would advise waiting a day or two to use it, as
I'm about to pull a large number of bugfixes into it (I'll update this
thread and provide a link once I do so).
Christian, the index-helper/watchman series are posted because you
showed interest in this area. I'm not rerolling to address David's
comments on the series for now. Take your time evaluate the two
approaches, then you can pick one (and let me know if you want me to
hand my series over, very glad to do so).
--
Duy
From: Christian Couder <hidden> Date: 2016-06-15 23:07:09
On Tue, Nov 3, 2015 at 6:45 AM, Duy Nguyen [off-list ref] wrote:
On Mon, Nov 2, 2015 at 9:56 PM, David Turner [off-list ref] wrote:
quoted
On Thu, 2015-10-29 at 09:10 +0100, Christian Couder wrote:
quoted
quoted
We're using Watchman at Twitter. A week or two ago posted a dump of our
code to github, but I would advise waiting a day or two to use it, as
I'm about to pull a large number of bugfixes into it (I'll update this
thread and provide a link once I do so).
Christian, the index-helper/watchman series are posted because you
showed interest in this area. I'm not rerolling to address David's
comments on the series for now.
Ok no problem. Thanks a lot to you and David for posting your rebased series!
Take your time evaluate the two
approaches, then you can pick one (and let me know if you want me to
hand my series over, very glad to do so).
From: David Turner <hidden> Date: 2016-06-15 23:07:10
On Tue, 2015-11-03 at 08:09 +0100, Christian Couder wrote:
On Tue, Nov 3, 2015 at 6:45 AM, Duy Nguyen [off-list ref] wrote:
quoted
On Mon, Nov 2, 2015 at 9:56 PM, David Turner [off-list ref] wrote:
quoted
On Thu, 2015-10-29 at 09:10 +0100, Christian Couder wrote:
quoted
quoted
We're using Watchman at Twitter. A week or two ago posted a dump of our
code to github, but I would advise waiting a day or two to use it, as
I'm about to pull a large number of bugfixes into it (I'll update this
thread and provide a link once I do so).
Christian, the index-helper/watchman series are posted because you
showed interest in this area. I'm not rerolling to address David's
comments on the series for now.
Ok no problem. Thanks a lot to you and David for posting your rebased series!
quoted
Take your time evaluate the two
approaches, then you can pick one (and let me know if you want me to
hand my series over, very glad to do so).
Yeah, I will do that, thanks again!
To be clear: I think Duy's approach is probably best in the long term.