David Turner [off-list ref] writes:
I've revived and modified Ronnie Sahlberg's work on the refs db
backend.
The work is on top of be3c13e5564, Junio's "First batch for 2.5 cycle".
I recognize that there have been changes to the refs code since then,
and that there are some further changes in-flight from e.g. Michael
Haggerty. If there is interest in this, I can rebase once Michael's
changes land.
...
The db backend runs git for-each-ref about 30% faster than the files
backend with fully-packed refs on a repo with ~120k refs. It's also
about 4x faster than using fully-unpacked refs. In addition, and
perhaps more importantly, it avoids case-conflict issues on OS X.
I chose to use LMDB for the database...
...
Ronnie Sahlberg's original version of this patchset used tdb. The
advantage of tdb is that it's smaller (~125k). The disadvantages are
that tdb is hard to build on OS X. It's also not in homebrew. So lmdb
seemed simpler.
"If there is interest"? Shut up and take my money ;-)
More seriously, that's great that you stepped up to resurrect this
topic. In a sense, the choice of sample database backend does not
matter. I do not care if it is tdb, lmdb, or even Berkeley DB as
long as it functions. ;-)
As long as the interface between ref-transaction system on the Git
side and the database backend is designed right, your lmdb thing can
serve as a reference implementation for other people to plug other
database backends to the same interface, right? As one step to
validate the interface to the database backends, it would be nice to
eventually have at least two backends that talk to meaningfully
different systems, but we have to start somewhere, and "for now we
have lmdb" is as good a place to start as any other db backend.
I wonder if we can do a "filesystem" backend on top of the same
backend interface---is that too much impedance mismatch to make it
unpractical?
Thanks.
On Tue, Jun 23, 2015 at 12:36 PM, Junio C Hamano [off-list ref] wrote:
"If there is interest"? Shut up and take my money ;-)
Yeah. This may be the next big thing since pack bitmap. It's even
better if it enters 'master' hand in hand with pack protocol v2, but I
think v2 needs more time.
On Tue, Jun 23, 2015 at 7:50 AM, David Turner [off-list ref] wrote:
To test this backend's correctness, I hacked test-lib.sh and
test-lib-functions.sh to run all tests under the refs backend.
Now we have two. split-index also benefits from running through full
test suite like this. I propose we make "make test" run the test suite
twice. The first run is with default configuration, no split index, no
fancy ref backend. The second run enables split-index and switches to
new backend, running through all test cases. In future we can also
enable packv4 in this second run. There won't be a third run.
When the second ref backend comes, we can switch between the two
backends using a random number generator where we control both
algorithm and seed, so that when a test fails, the user can give us
their seed and we can re-run with the same configuration.
Dozens of tests use manual ref/reflog reading/writing, or create submodules
without passing --refs-backend-type to git init. If those tests are
changed to use the update-ref machinery or test-refs-be-db (or, in the
case of packed-refs, corrupt refs, and dumb fetch tests, are skipped),
the only remaining failing tests are the git-new-workdir tests and the
gitweb tests.
I haven't read the series, but I guess you should also add a few tests
to run on the first run, so new code is exercised a bit even if people
skip the second run.
--
Duy
On Mon, 2015-06-22 at 22:36 -0700, Junio C Hamano wrote:
David Turner [off-list ref] writes:
quoted
I've revived and modified Ronnie Sahlberg's work on the refs db
backend.
The work is on top of be3c13e5564, Junio's "First batch for 2.5 cycle".
I recognize that there have been changes to the refs code since then,
and that there are some further changes in-flight from e.g. Michael
Haggerty. If there is interest in this, I can rebase once Michael's
changes land.
...
The db backend runs git for-each-ref about 30% faster than the files
backend with fully-packed refs on a repo with ~120k refs. It's also
about 4x faster than using fully-unpacked refs. In addition, and
perhaps more importantly, it avoids case-conflict issues on OS X.
I chose to use LMDB for the database...
...
Ronnie Sahlberg's original version of this patchset used tdb. The
advantage of tdb is that it's smaller (~125k). The disadvantages are
that tdb is hard to build on OS X. It's also not in homebrew. So lmdb
seemed simpler.
"If there is interest"? Shut up and take my money ;-)
More seriously, that's great that you stepped up to resurrect this
topic. In a sense, the choice of sample database backend does not
matter. I do not care if it is tdb, lmdb, or even Berkeley DB as
long as it functions. ;-)
As long as the interface between ref-transaction system on the Git
side and the database backend is designed right, your lmdb thing can
serve as a reference implementation for other people to plug other
database backends to the same interface, right?
Yes.
As one step to
validate the interface to the database backends, it would be nice to
eventually have at least two backends that talk to meaningfully
different systems, but we have to start somewhere, and "for now we
have lmdb" is as good a place to start as any other db backend.
I wonder if we can do a "filesystem" backend on top of the same
backend interface---is that too much impedance mismatch to make it
unpractical?
The patch series does include a filesystem backend, which is simply the
current ref infrastructure with extremely minor changes.
On Tue, 2015-06-23 at 17:23 +0700, Duy Nguyen wrote:
On Tue, Jun 23, 2015 at 7:50 AM, David Turner
[off-list ref] wrote:
quoted
To test this backend's correctness, I hacked test-lib.sh and
test-lib-functions.sh to run all tests under the refs backend.
Now we have two. split-index also benefits from running through full
test suite like this. I propose we make "make test" run the test suite
twice. The first run is with default configuration, no split index, no
fancy ref backend. The second run enables split-index and switches to
new backend, running through all test cases. In future we can also
enable packv4 in this second run. There won't be a third run.
When the second ref backend comes, we can switch between the two
backends using a random number generator where we control both
algorithm and seed, so that when a test fails, the user can give us
their seed and we can re-run with the same configuration.
I'm not in love with this idea, because it makes it hard to do
exhaustive testing efficiently. I would rather have make test run
through all tests under all combinations -- or at least all relevant
tests. We could perhaps mark tests with a list of features that they
exercise, so that we don't have to run e.g. t8xxx with alternate refs
backends.
Dozens of tests use manual ref/reflog reading/writing, or create
submodules
quoted
without passing --refs-backend-type to git init. If those tests are
changed to use the update-ref machinery or test-refs-be-db (or, in
the
quoted
case of packed-refs, corrupt refs, and dumb fetch tests, are
skipped),
quoted
the only remaining failing tests are the git-new-workdir tests and
the
quoted
gitweb tests.
I haven't read the series, but I guess you should also add a few tests
to run on the first run, so new code is exercised a bit even if people
skip the second run.
I did this already, yes.