From: Junio C Hamano <hidden> Date: 2016-06-15 22:41:59
quoted
quoted
quoted
quoted
"JM" == Jason McMullan [off-list ref] writes:
JM> Sorry about being a pest, but this worries me. Please assuage my fears.
Earlier I said I suspected that the original code mishandled
recovery from a botched tree/commit dependent transfer, but that
was not the case. The last test in the new test script I added
in the patch you are responding to covers that case.
JM> (Or, if you'd like, I can rework pull.c to use the
JM> verification-before-store technique I used in my git-daemon patch, so
JM> all the *-pull mechanisms will be 'safe')
I would appreciate the offer. I, however, would have to warn
you that the "problem" lies in the way the current pull
structure devides responsibility between the pull.c and transfer
backends. The pull.c implements the dependency logic, and
transfer backends are to populate the database while being
oblivious of that logic. From the purist point of view (I am
sympathetic to your "place only the verified objects in the
database" principle), I am not entirely happy with that
division, but at the same time I understand why it is done that
way and even like it from practical standpoint. Otherwise you
need to keep a bunch of objects somewhere outside the database
along with the list of "things to rename to the final database
name when we are done". You would somehow need to do clean-up
when we fail in the middle _anyway_.
In other words, the current structure is optimized for non-
failure case, as it should be. The original implementation
(credit goes to Dan Barkalow) knew how to recover from failed
transfer (including the case that you first pull with -c or -t
without using -a to miss some required objects to satisfy -a) by
simply running pull again, and with the --recover flag, it now
knows how to recover from failed deltified object transfer as
well.
From: Daniel Barkalow <hidden> Date: 2016-06-15 22:41:59
On Sun, 5 Jun 2005, Junio C Hamano wrote:
quoted
quoted
quoted
quoted
quoted
"JM" == Jason McMullan [off-list ref] writes:
JM> Sorry about being a pest, but this worries me. Please assuage my fears.
Earlier I said I suspected that the original code mishandled
recovery from a botched tree/commit dependent transfer, but that
was not the case. The last test in the new test script I added
in the patch you are responding to covers that case.
It does the O(history) method for correctness at the expense of
efficiency; my hope is that a bit of caching can fix the efficiency issue
as well. So the question is not really "not safe" as "slow". Of course, it
takes a while for this to become an issue, given the relationship of
remote access latency to local access bandwidth. That is, you need a
really big history and to be getting very little new data before you'll
complain.
JM> (Or, if you'd like, I can rework pull.c to use the
JM> verification-before-store technique I used in my git-daemon patch, so
JM> all the *-pull mechanisms will be 'safe')
I would appreciate the offer. I, however, would have to warn
you that the "problem" lies in the way the current pull
structure devides responsibility between the pull.c and transfer
backends. The pull.c implements the dependency logic, and
transfer backends are to populate the database while being
oblivious of that logic. From the purist point of view (I am
sympathetic to your "place only the verified objects in the
database" principle), I am not entirely happy with that
division, but at the same time I understand why it is done that
way and even like it from practical standpoint.
At one point I'd written a patch that split out the tmpfile usage of
write_sha1_file(), made the filenames predictable, and used it for
everything that writes those files. It had an "open" part and a
"close" part (where the close also moved the file into place). This would
give the code better atomicity and protect against races between reading
and validation. On the other hand, there's no reason to use an anonymous
temp file; just <filename>.partial or similar (with the proper open
flags) would be sufficient and easier to clean or commit. Note that we
want to support /tmp and the object directory being on different
filesystems, also. (And all the open and place logic is nicely wrapped up
in sha1_file.c)
Aside from the question of whether we want to insist that the object
database only includes objects such that everything reachable is also
present, we certainly want to only include objects which we have
completely fetched, which are generically well-formed, and which have the
advertized hash, and having there never be an unvalidated file at the
filename would be good.
By this reasoning, a file should only be renamed after all of the delta
requirements are satisfied, but before tree and commit requirements are
satisfied. We certainly aren't going to have much use for files whose
contents we cannot get. This means that we'd like to have multiple
unplaced files, but we don't need to read the contents of an unplaced
file.
-Daniel
*This .sig left intentionally blank*
From: McMullan, Jason <hidden> Date: 2016-06-15 22:41:59
Subject Was: [PATCH] pull: gracefu[PAlly recover from delta retrieval
failure.]
[snip lots of really good information about the thinking
behind the design of the pull mechanisms ]
Ok, so would I be correct in the following assumptions
about the validity of a 'consistent' .git/objects database:
============================================================
Commits:
* May have the tree they refer to in the database
* Must have their parents in the database
Trees:
* Must have the blobs they refer to in the database
* Must have the trees they refer to in the database
Deltas:
* Must have the referred to object in the database
Blobs:
* No references to check
============================================================
In short, the database would contain:
* The entire commit history
* Selected commits would have the entire tree available
Correct, or totally mistaken? If mistaken, what are the consitency
rules?
[Oh, and does PGP signing my messages bug anybody? If so, I can stop
doing that on this list]
--
Jason McMullan [off-list ref]
TimeSys Corporation
From: Daniel Barkalow <hidden> Date: 2016-06-15 22:41:59
On Mon, 6 Jun 2005, McMullan, Jason wrote:
Subject Was: [PATCH] pull: gracefu[PAlly recover from delta retrieval
failure.]
[snip lots of really good information about the thinking
behind the design of the pull mechanisms ]
Ok, so would I be correct in the following assumptions
about the validity of a 'consistent' .git/objects database:
============================================================
Commits:
* May have the tree they refer to in the database
* Must have their parents in the database
May have their parents in the database; we want to be able to drop ancient
history from non-archival sites at some point, if nothing else.
Trees:
* Must have the blobs they refer to in the database
* Must have the trees they refer to in the database
It's probably true that there's no point to having a tree available if you
don't have its contents, although that's a convenient intermediate stage,
so that you can look up the contents of the tree with the ordinary parsing
code. On the other hand, I could imagine an ARM developer completely
ignoring arch/i386 (and just having write-tree use the parent tree's value
for it).
Deltas:
* Must have the referred to object in the database
Yes. Can't unpack without them.
Blobs:
* No references to check
Right.
Also, tags reference objects of unknown type; it's probably not vital to
have the object.
My bias is to call a database consistent with only deltas having the
referents; the rest goes towards completeness, since you have and can read
everything that you have anything for (but may not be able to do some
particular operation).
-Daniel
*This .sig left intentionally blank*
From: McMullan, Jason <hidden> Date: 2016-06-15 22:41:59
On Mon, 2005-06-06 at 12:21 -0400, Daniel Barkalow wrote:
[snip snip]
My bias is to call a database consistent with only deltas having the
referents; the rest goes towards completeness, since you have and can read
everything that you have anything for (but may not be able to do some
particular operation).
Now, if we had consistent URIs for the .git/branches/* files, we could
do 'lazy-pull' and really have our cake and eat it too.
--
Jason McMullan [off-list ref]
TimeSys Corporation