From: Junio C Hamano <hidden> Date: 2016-06-15 22:42:17
Shallow History Cloning
=======================
One good thing about git repository is that each clone is a
freestanding and complete entity, and you can keep developing in
it offline, without talking to the outside world, knowing that
you can sync with them later when online.
It is also a bad thing. It gives people working on projects
with long development history stored in CVS a heart attack when
we tell them that their clones need to store the whole history.
There was a suggestion by Linus to allow a partial clone using a
syntax like this:
$ git clone --since=v2.6.14 git://.../linux-2.6/ master
Here is an outline of what changes are needed to the current
core to do this.
Strategy
--------
We have `info/grafts` mechanism to fake parent information for
commit objects. Using this facility, we could roughly do:
. Download the full tree for v2.6.14 commit and store its
objects locally.
. Set up `info/grafts` to lie to the local git that Linux kernel
history began at v2.6.14 version.
. Run `git fetch git://.../linux-2.6 master`, with a local ref
pointing at v2.6.14 commit, to pretend that we have everything
up to v2.6.14 to `upload-pack` running on the other end.
. Update the `origin` branch with the master commit object name
we just fetched from Linus.
There are some issues.
. In the fetch above to obtain everything after v2.6.14, and
future runs of `git fetch origin`, if a blob that is in the
commit being fetched happens to match what used to be in a
commit that is older than v2.6.14 (e.g. a patch was reverted),
`upload-pack` running on the other end is free to omit sending
it, because we are telling it that we are up to date with
respect to v2.6.14. Although I think the current `rev-list
--objects` implementation does not always do such a revert
optimization if the revert is to a blob in a revision that is
sufficiently old, it is free to optimize more aggressively in
the future.
. Later when the user decides to fetch older history, the
operation can become a bit cumbersome.
I think the latter one is cumbersome but is doable -- we could
do the equivalent of:
$ git clone --since=v2.6.13 origin v2.6.14
place all the objects obtained by such a clone/fetch operation
and remember that now we have history beginning at v2.6.13. So
let's worry about that later.
For the first issue, we need to have the other end cooperate
while fetching from it. If the other end also thinks the
development started at v2.6.14, even if we tell that we have the
history up to v2.6.14 (or a commit we obtained since then),
there is no way for `upload-pack` running there to optimize too
agressively and assume we have a blob that appeared in v2.6.13.
More simply, we do not have to tell them we have anything -- if
the other end thinks the epoch is at v2.6.14, only commits that
comes later will be sent to us.
Design
------
First, to bootstrap the process, we would need to add a way to
obtain all objects associated with a commit. We could do a new
program, or we could implement this as a protocol extension to
`upload-pack`. My current inclination is the latter.
When talking with `upload-pack` that supports this extension,
the downloader can give one commit object name and get a pack
that contains all the objects in the tree associated with that
commit, plus the commit object itself. This is a rough
equivalent of running the commit walker with the `-t` flag.
Another functionality we would need is to tell `upload-pack` to
use `info/grafts` of downloader's choice. With this, after
fetching the objects for v2.6.14 commit, the downloader can set
up its own grafts file to cauterize the development history at
v2.6.14, and tell the `upload-pack` to pretend the kernel
history starts at that commit, while sending the tip of Linus'
development track to us.
Using the extended protocol (let's call it 'shallow' extension),
a clone to create a repository that has only recent kernel
history since v2.6.14 goes like this:
The first client is to fetch the v2.6.14 itself.
[NOTE]
Most likely this is not directly run by the user but is run as
the first command invoked by the shallow clone script.
1. The `fetch-pack` command acquires a new option, `--single`:
$ git-fetch-pack --single git://.../linux-2.6/ v2.6.14
This talks with `upload-pack` on the kernel.org server via
`git-daemon`.
2. `upload-pack` tells the fetcher what commits it has,
what their refs are, and what protocol extensions it
supports, as usual.
3. If it does not see `shallow` extension supported, there is no
way to get a single tree, so things fail here. Otherwise, it
sends `single X{40}\0` request, instead of the usual `want`
line. The object name sent here is the desired commit.
4. `upload-pack` notices this is a single commit request, and
sends an ACK if it can satisfy the request (or a NAK if it
can't, e.g. it does not have the asked commit). Instead of
doing the usual `get_common_commits` followed by
`create_pack_file`, it does:
$ git rev-list -n1 --objects $commit | git pack-object
and sends the result out.
5. The fetcher checks the ACK and receives the objects.
After the above exchange, we have downloaded v2.6.14 commit and
its objects but not its history. `git-fetch-pack` would output
the tag object name for `v2.6.14` and we would stash it away in
`$GIT_DIR/FETCH_HEAD` as usual. Then we set up `info/grafts`
with this:
$ git rev-parse FETCH_HEAD^{commit} >"$GIT_DIR/info/grafts"
This cauterizes the history on our end.
The second phase of the shallow clone is to fetch the history
since v2.6.14 to the tip.
1. The `fetch-pack` command is run as usual. Most likely the
command line run by the shallow clone script would be:
$ git fetch-pack git://.../linux-2.6/ master
Notice there is nothing magical about it. It is just the
business as usual.
2. `upload-pack` does its usual greeting to the downloader.
3. We notice `shallow` extension again, and first send out
`graft X{40}\0` request. The syntax of graft request would
be `graft ` followed by one or more commit object names on a
line separated with SP. After sending out all the needed
graft requests (in this example there is only one, to
cauterize the history at v2.6.14), it does the usual `want
X{40}\0multi_ack` and a flush.
4. `upload-pack` notices graft requests, reinitializes its graft
information with what it receives from the other end, and
then records `want`.
5. After the above steps, the usual `upload-pack` vs
`fetch-pack` exchange continues and objects needed to
complete the Linus' tip of development trail for somebody who
has v2.6.14 are sent in a pack. The difference from the
usual operation is that `upload-pack` during this run thinks
v2.6.14 commit does not have any parent.
The exact sequence from the second part of the initial "shallow
clone" can be used for further updates.
There is a small issue about the actual implementation. In the
above description I pretended that `upload-pack` can be told to
use phony grafts information, but in the current implementation
the program that needs to use phony grafts information is
`rev-list` spawned from it. We _could_ point GIT_GRAFT_FILE
environment variable point at a temporary file while we do so,
but I'd like to avoid using a temporary file if possible, given
that `upload-pack` is run from `git-daemon`. Maybe we could
give --read-graft-from-stdin flag to `rev-list` for this
purpose.
Anybody want to try?
From: Johannes Schindelin <hidden> Date: 2016-06-15 22:42:17
Hi,
On Sun, 29 Jan 2006, Junio C Hamano wrote:
Strategy
--------
We have `info/grafts` mechanism to fake parent information for
commit objects. Using this facility, we could roughly do:
. Download the full tree for v2.6.14 commit and store its
objects locally.
On first read, I mistook "tree" for "commit"...
. Set up `info/grafts` to lie to the local git that Linux kernel
history began at v2.6.14 version.
Maybe also record this in .git/config, so that you can
- disallow fetching from this repo, and
- easily extend the shallow copy to a larger shallow one, or a full one.
. Run `git fetch git://.../linux-2.6 master`, with a local ref
pointing at v2.6.14 commit, to pretend that we have everything
up to v2.6.14 to `upload-pack` running on the other end.
How about refs/tags/start_shallow?
. Update the `origin` branch with the master commit object name
we just fetched from Linus.
Design
------
[...]
Another functionality we would need is to tell `upload-pack` to
use `info/grafts` of downloader's choice. With this, after
fetching the objects for v2.6.14 commit, the downloader can set
up its own grafts file to cauterize the development history at
v2.6.14, and tell the `upload-pack` to pretend the kernel
history starts at that commit, while sending the tip of Linus'
development track to us.
Why not just start another fetch? Then, "have <refs/tags/start_shallow>"
would be sent, and upload-pack does the right thing?
If you absolutely want to get only one pack, which then is stored as-is,
upload-pack could start two rev-list processes: one for the tree and one
for all the rest.
[...]
[NOTE]
Most likely this is not directly run by the user but is run as
the first command invoked by the shallow clone script.
Better make it an option to git-clone
4. `upload-pack` notices this is a single commit request, and
sends an ACK if it can satisfy the request (or a NAK if it
can't, e.g. it does not have the asked commit). Instead of
doing the usual `get_common_commits` followed by
`create_pack_file`, it does:
$ git rev-list -n1 --objects $commit | git pack-object
Here it could say
(git rev-list -n1 --objects $commit_since; git rev-list --objects
^$commit_since $commit) | git pack-object
If the former is still needed (e.g. for git-tar-remote-tree), we could
distinguish "single <ref>" and "shallow <ref>" commands.
[...]
The second phase of the shallow clone is to fetch the history
since v2.6.14 to the tip.
As I outlined above, I don't see the need for this.
Ciao,
Dscho
From: Simon Richter <hidden> Date: 2016-06-15 22:42:17
Hi,
Johannes Schindelin wrote:
quoted
. Set up `info/grafts` to lie to the local git that Linux kernel
history began at v2.6.14 version.
Maybe also record this in .git/config, so that you can
I like that "config" thing less and less every day. It appears to become
a kind of registry, where having dedicated files for specific
functionality would provide the robustness of tools not having to touch
things they do not care about; but that's just personal opinion.
- disallow fetching from this repo, and
Why? It's perfectly acceptable to pull from an incomplete repo, as long
as you don't care about the old history.
- easily extend the shallow copy to a larger shallow one, or a full one.
Hrm, I think there should also be a way to shrink a repo and "forget"
old history occasionally (obviously, use of that feature would be highly
discouraged).
quoted
. Run `git fetch git://.../linux-2.6 master`, with a local ref
pointing at v2.6.14 commit, to pretend that we have everything
up to v2.6.14 to `upload-pack` running on the other end.
How about refs/tags/start_shallow?
No, as that would imply that cloning from such a repo is disallowed.
IMO, it may be a lot more robust to just have a list of "cutoff" object
ids in .git/shallow instead of messing with grafts here, as adding or
removing a line from that file is an easier thing to do for porcelain
(or by hand) than rewriting the grafts file. Whether that list would be
inclusive or exclusive would need to be decided still.
Simon
From: Johannes Schindelin <hidden> Date: 2016-06-15 22:42:17
Hi,
On Mon, 30 Jan 2006, Simon Richter wrote:
Johannes Schindelin wrote:
quoted
quoted
. Set up `info/grafts` to lie to the local git that Linux kernel
history began at v2.6.14 version.
quoted
Maybe also record this in .git/config, so that you can
I like that "config" thing less and less every day. It appears to become a
kind of registry, where having dedicated files for specific functionality
would provide the robustness of tools not having to touch things they do not
care about; but that's just personal opinion.
It is becoming sort of a registry: it contains metadata about the current
repository, easily available to scripts and programs.
I beg to differ on your personal opinion on the grounds that the
robustness comes from testing, not from diversity. I much prefer to have a
well tested config mechanism to having dozens of differently formatted
files with less-than-well tested parsers.
Thank you for the insights in your personal opinion anyway.
quoted
- disallow fetching from this repo, and
Why? It's perfectly acceptable to pull from an incomplete repo, as long as you
don't care about the old history.
Right. But should that be the default? I don't think so. Therefore:
disable it, and if the user is absolutely sure to do dumb things, she'll
have to enable it explicitely.
quoted
- easily extend the shallow copy to a larger shallow one, or a full one.
Hrm, I think there should also be a way to shrink a repo and "forget" old
history occasionally (obviously, use of that feature would be highly
discouraged).
Yes. And you need information about how shallow it used to be. My
suggestion was to store that information at a place specific to that
repository (see above).
quoted
quoted
. Run `git fetch git://.../linux-2.6 master`, with a local ref
pointing at v2.6.14 commit, to pretend that we have everything
up to v2.6.14 to `upload-pack` running on the other end.
quoted
How about refs/tags/start_shallow?
No, as that would imply that cloning from such a repo is disallowed.
See above.
IMO, it may be a lot more robust to just have a list of "cutoff" object ids in
.git/shallow instead of messing with grafts here, as adding or removing a line
from that file is an easier thing to do for porcelain (or by hand) than
rewriting the grafts file. Whether that list would be inclusive or exclusive
would need to be decided still.
The functionality of cutoff objects is included in grafts functionality,
so why should we spend time on reimplementing a subset of features?
IMHO, adding and removing lines from scripts is fragile.
I beg your pardon, you want to edit this information *by hand*? Wow.
Ciao,
Dscho
From: Simon Richter <hidden> Date: 2016-06-15 22:42:17
Hi,
Johannes Schindelin wrote:
[config as a registry]
It is becoming sort of a registry: it contains metadata about the current
repository, easily available to scripts and programs.
Provided you have a parser that can handle it.
I beg to differ on your personal opinion on the grounds that the
robustness comes from testing, not from diversity. I much prefer to have a
well tested config mechanism to having dozens of differently formatted
files with less-than-well tested parsers.
Indeed. But we already have a method for associating data values with
keys in a hierarchical namespace, and that one is pretty well tested. :-)
quoted
Why? It's perfectly acceptable to pull from an incomplete repo, as long as you
don't care about the old history.
Right. But should that be the default? I don't think so. Therefore:
disable it, and if the user is absolutely sure to do dumb things, she'll
have to enable it explicitely.
What harm is done if I have an incomplete repository? It would probably
make more sense to emit a warning on clone and explain things if the
user tries to go to a version she doesn't have.
quoted
Hrm, I think there should also be a way to shrink a repo and "forget" old
history occasionally (obviously, use of that feature would be highly
discouraged).
Yes. And you need information about how shallow it used to be. My
suggestion was to store that information at a place specific to that
repository (see above).
Indeed, but you are keeping this information in two places, namely the
grafts file and the config file. This is asking for trouble if they ever
get out of sync.
quoted
quoted
How about refs/tags/start_shallow?
quoted
No, as that would imply that cloning from such a repo is disallowed.
See above.
Well, I can however see the use case of a developer hosting an
incomplete repo on a free web service and another developer wanting to
merge her changes into her (complete) repo. You would have to
specialcase this tag in the fetch operation to avoid copying it over.
What's probably worse: You can only have a single cutoff point that way.
You probably want multiple in case you want to cut off at a place where
development happened in multiple branches that got subsequently merged
inside the window of objects you keep.
The functionality of cutoff objects is included in grafts functionality,
so why should we spend time on reimplementing a subset of features?
I would ask for the grafts parser to add "fake" grafts when it
encounters the "shallow" file. Otherwise, it would be hard to
distinguish between grafts the user made when doing interesting merges,
and grafts that were created to build a shallow repo, because you would
need some heuristics to figure out the latter from the former if you
want to have a function in your porcelain to "pull more/all objects".
I beg your pardon, you want to edit this information *by hand*? Wow.
Yes. That is actually the reason I like git so much: I can repair it by
hand if something breaks, and this can be done with simple commands. I
can remove an object id from a file with "grep -v" or perl. I would need
to fire up an editor or hack a longer script if I wanted to fix
something inside a complex file that does multiple things.
Simon
From: Junio C Hamano <hidden> Date: 2016-06-15 22:42:17
Johannes Schindelin [off-list ref] writes:
quoted
. Download the full tree for v2.6.14 commit and store its
objects locally.
On first read, I mistook "tree" for "commit"...
It turns out that this 'single' request step is unneeded, as
long as we implement 'graft' requests. We can then tell
"Cauterize at v2.6.14 and give me the master" to `upload-pack`.
`upload-pack` would run `rev-list --objects master`, tries to include
everything that is reachable from "master", but notices that the
v2.6.14 commit does not have any parent (thanks to the
customized graft) and stops there -- the result is the history
since v2.6.14.
quoted
. Set up `info/grafts` to lie to the local git that Linux kernel
history began at v2.6.14 version.
Maybe also record this in .git/config, so that you can
- disallow fetching from this repo, and
- easily extend the shallow copy to a larger shallow one, or a full one.
I thought about that before I wrote the message, but it boils
down to grepping lines from grafts that have only one object
name (i.e. cauterizing records), so it is redundant.
Also there is no strict reason to forbid cloning from such a
shallow repository. No harm is done as long as you make it
clear to somebody who clones from you that what you have is a
shallow copy, so that the cloned repository can cauterize
history at appropriate places.
A second generation clone, when cloning from a shallow
repository, needs to mark itself that it has the same or
shallower history (otherwise a third generation clone from it
would not work), so the `upload-pack` protocol needs to be
updated to send grafts information the `upload-pack` side
usually uses to the downloader even when 'graft' request is not
used by the downloader. But once it is done, you should be able
to clone safely from a shallow repository and end up with a
repository with the same (or shallower -- if you asked to make a
shallow clone from it) history.
Why not just start another fetch? Then, "have <refs/tags/start_shallow>"
would be sent, and upload-pack does the right thing?
Yes, almost. We need to realize that `upload-pack` that hears
"have A, want B" is allowed to omit objects that appear in
`ls-tree B` output but not in `ls-tree A`. "have A" means not
just "I have A", but "I have A and all of its ancestors", so
just sending "have start_shallow" (or start_shallow^ for that
matter) is not quite enough [*1*].
If you absolutely want to get only one pack, which then is stored as-is,
upload-pack could start two rev-list processes: one for the tree and one
for all the rest.
The message you are responding did two separate transfers (one
'single', and another 'fetch'); I do not particularly mind doing
two (it is just an initial clone anyway), but as I said it turns
out that we do not need the initial 'single'.
quoted
[NOTE]
Most likely this is not directly run by the user but is run as
the first command invoked by the shallow clone script.
Better make it an option to git-clone
Probably -- I was just outlining the lowest-level mechanism and
haven't thought much about the UI.
[Footnote]
*1* This is true even without more aggressive optimization by
rev-list that does not exist there yet. Here is a minimalistic
demonstration. One file project with a handful straight-line
commits. Each change to the file reverts the change made by the
previous commit.
* The HEAD commit has "white", the HEAD~1 "black" and HEAD~2
"white".
* We say we are interested in things since HEAD~2 (i.e. we
pretend that the history starts at HEAD~1 and it does not
have a parent) and ask for HEAD.
* Notice that only one copy of the file appears in the output.
It is "black" blob. We do not get "white" blob because we
are telling it that we _have_ HEAD~2. The resulting set of
objects is not enough to check-out the HEAD commit.
This roughly corresponds to your "have shallow_start", but not
quite -- in that sequence you have objects for HEAD~2 commit.
But the point is that I want to leave the door open for
optimizing upload-pack, so that it can choose to omit objects
that do not appear in A when you say "have A", if the object
appears in one of A's ancestors.
-- >8 --
#!/bin/sh
rm -fr .git
git init-db
zebra=white
echo $zebra >file
git add file
git commit -m initial
for i in 0 1 2 3 4 5
do
case $zebra in
white) zebra=black ;;
black) zebra=white ;;
esac
echo $zebra >file
git commit -a -m "$i $zebra"
done
git rev-list --objects HEAD~2..HEAD |
git name-rev --stdin
From: Junio C Hamano <hidden> Date: 2016-06-15 22:42:17
Simon Richter [off-list ref] writes:
quoted
- disallow fetching from this repo, and
Why? It's perfectly acceptable to pull from an incomplete repo, as
long as you don't care about the old history.
I agree. As long as the cloned one can record itself as a
shallow one (and with what epochs), I do not see a reason to
forbid second generation clone from a shallow repository.
Hrm, I think there should also be a way to shrink a repo and "forget"
old history occasionally (obviously, use of that feature would be
highly discouraged).
I do not think of a reason to discourage it, and I think you can
do the "forgetting" part with the current set of tools. Choose
appropriate cauterizing points, set up info/grafts and running
"repack -a -d" would be sufficient.
IMO, it may be a lot more robust to just have a list of "cutoff"
object ids in .git/shallow instead of messing with grafts here, as
adding or removing a line from that file is an easier thing to do for
porcelain (or by hand) than rewriting the grafts file. Whether that
list would be inclusive or exclusive would need to be decided still.
I would rather not to have .git/shallow nor .git/shallow_start.
Cauterizing is not any more special than other grafts entries.
If you have grafted historical kernel repository behind the
official kernel repository with 2.6.12-rc2 epoch, I do not think
of any reason to forbid people from cloning such with the
grafts.
From: Junio C Hamano <hidden> Date: 2016-06-15 22:42:17
Johannes Schindelin [off-list ref] writes:
quoted
quoted
- disallow fetching from this repo, and
Why? It's perfectly acceptable to pull from an incomplete
repo, as long as you don't care about the old history.
Right. But should that be the default? I don't think so. Therefore:
disable it, and if the user is absolutely sure to do dumb things, she'll
have to enable it explicitely.
If the downstream person wants to have a shallow history of post
X.org X server core to further hack on it, I do not think of a
reason why we would want to refuse her from cloning a repository
of a fellow developer who has already done such a shallow copy.
If such a clone is done without telling the downstream that the
result is a shallow one, it is "dumb". I would agree it should
not be done. We need to propagate the grafts to the downstream
when a clone is done because of this.
By the way, please refrain from discussing .git/config vs
.git/eparate-config-files issue in this thread. My personal
feeling so far is that the information current graft represents
is good enough to support shallow clones, and if not we can
extend its semantics to support such. It can be discussed
independently if it is a good idea to move the final result
(grafts with updated semantics) to config file. Even if we end
up not doing any of the shallow cloning support we have been
discussing, moving the information in .git/info/grafts to config
might make sense. The issue is tangential.
Why? It's perfectly acceptable to pull from an incomplete repo, as
long as you don't care about the old history.
I agree. As long as the cloned one can record itself as a
shallow one (and with what epochs), I do not see a reason to
forbid second generation clone from a shallow repository.
I agree too
Cauterizing is not any more special than other grafts entries.
If you have grafted historical kernel repository behind the
official kernel repository with 2.6.12-rc2 epoch, I do not think
of any reason to forbid people from cloning such with the
grafts.
I built my public repository from a cautorized one and everybody who
is pulling from mine is aware of the lack of the full history but they
actually don't care. If someone is pulling from my repo, he actually
wants to work on my project which do not need any old thing...
Thanks
--
Franck
From: Johannes Schindelin <hidden> Date: 2016-06-15 22:42:17
Hi,
On Mon, 30 Jan 2006, Junio C Hamano wrote:
Johannes Schindelin [off-list ref] writes:
quoted
quoted
quoted
- disallow fetching from this repo, and
Why? It's perfectly acceptable to pull from an incomplete
repo, as long as you don't care about the old history.
Right. But should that be the default? I don't think so. Therefore:
disable it, and if the user is absolutely sure to do dumb things, she'll
have to enable it explicitely.
If the downstream person wants to have a shallow history of post
X.org X server core to further hack on it, I do not think of a
reason why we would want to refuse her from cloning a repository
of a fellow developer who has already done such a shallow copy.
Okay. But in their case, they'll probably do what was done with Linux:
start afresh. If you want to have the old history, you can import it and
merge it via a graft.
If such a clone is done without telling the downstream that the
result is a shallow one, it is "dumb". I would agree it should
not be done.
That was my point. As long as you don't make sure the client handles the
shallow upstream gracefully, it is dangerous. At the moment, there are too
many code parts relying on the completeness of the repository (local and
remote).
Since I wrote this, I realized that the problem I saw is not limited to
shallow upstream, but there is a subtle issue with shallow downstreams,
too:
Just imagine this: Alice starts a project, Bob makes a shallow copy from
it when Alice just reverted an experimental feature. Then, Alice decides
the experimental feature was not bad at all and reverts the revert. Bob
pulls from Alice: Alice's upload-pack assumes Bob already has the original
files (now re-reverted), and Bob ends up with a broken repository.
While writing the last paragraph, it became clear to me that the shallow
thing is very fragile: IMHO it is impossible to be fully backwards
compatible (remember: you should not force anybody to upgrade).
By the way, please refrain from discussing .git/config vs
.git/eparate-config-files issue in this thread.
Okay. I will shut up on that issue.
My personal feeling so far is that the information current graft
represents is good enough to support shallow clones, and if not we can
extend its semantics to support such.
No. The grafts are more powerful. I have quite a few repos here in which I
heavily work with grafts, and they are no cutoffs for shallow repos. They
are hard links between different lines of development. For example, I use
them to map merges in cvsimported projects, thus fixing a shortcoming of
CVS. Also, you can "add" history.
If you now rely on the grafts file to determine what was a cutoff, you may
well end up with bogus cutoffs.
Ciao,
Dscho
From: Simon Richter <hidden> Date: 2016-06-15 22:42:17
Hi,
Johannes Schindelin wrote:
quoted
If the downstream person wants to have a shallow history of post
X.org X server core to further hack on it, I do not think of a
reason why we would want to refuse her from cloning a repository
of a fellow developer who has already done such a shallow copy.
Okay. But in their case, they'll probably do what was done with Linux:
start afresh. If you want to have the old history, you can import it and
merge it via a graft.
Well, in the Linux case the problem was not knowing what the SHA1 sum of
the entire Linux history was. In the shallow repo case we know it, so
there is no point in throwing away that information.
quoted
If such a clone is done without telling the downstream that the
result is a shallow one, it is "dumb". I would agree it should
not be done.
That was my point. As long as you don't make sure the client handles the
shallow upstream gracefully, it is dangerous. At the moment, there are too
many code parts relying on the completeness of the repository (local and
remote).
Well, the important thing would be that commands that can work (a merge
only needs to find the most recent common ancestor, etc) do work, and
commands that cannot ("log") emit sensible diagnostics.
Just imagine this: Alice starts a project, Bob makes a shallow copy from
it when Alice just reverted an experimental feature. Then, Alice decides
the experimental feature was not bad at all and reverts the revert. Bob
pulls from Alice: Alice's upload-pack assumes Bob already has the original
files (now re-reverted), and Bob ends up with a broken repository.
I know far too little about the internal workings for that, but I'd
assume that in this case Bob's copy starts at the commit that was never
in question (and he never saw the reverted commit), and Alice's contains
a commit on top of that. That one should work. But the other way 'round
is problematic, when Bob starts with a commit that has been reverted in
Alice's repository. The solution is for Bob to ask Alice's repo for the
common ancestor of his shallow base and Alice's HEAD. Alice's repo can,
however, fail to deliver these if there has been a purge since, in that
case, stuff needs to be merged by hand (but you already have a problem
if someone clones your repo before you revert changes, so no regression
here).
If you now rely on the grafts file to determine what was a cutoff, you may
well end up with bogus cutoffs.
Exactly that was my concern earlier; my database design gut feeling
tells me that information duplication is not good either, hence my
suggestion to split off these grafts into a separate file in order to
mark them as cutoff points.
Simon
From: Johannes Schindelin <hidden> Date: 2016-06-15 22:42:17
Hi,
On Tue, 31 Jan 2006, Simon Richter wrote:
Well, the important thing would be that commands that can work (a merge only
needs to find the most recent common ancestor, etc) do work, and commands that
cannot ("log") emit sensible diagnostics.
No it would not.
A commit is a very small object which points (among others) to a tree
object.
A tree object corresponds to a directory (that is, it can point to a
number of tree and blob objects).
A blob object corresponds to a file (that is, git never parses its
contents).
If two separate revisions contain the same file (i.e. same contents), this
is not duplicated, but the corresponding tree objects point to the same
object.
If you pull, upload-pack will think you have *every* object depending on
every ref you have stored.
Say you have three revisions, A -> B -> C, and A and C contain the
same file bla.txt, and the client says it has B, the upstream upload-pack
assumes you have bla.txt.
I know far too little about the internal workings for that, [...]
I hope I clarified the important aspect.
quoted
If you now rely on the grafts file to determine what was a cutoff, you may
well end up with bogus cutoffs.
Exactly that was my concern earlier; my database design gut feeling tells me
that information duplication is not good either, [...]
You only have two choices: you proposed code duplication, and yours truly
proposed data duplication.
As is known from good database design: a few redundancies here and there
are typically needed for good performance.
Ciao,
Dscho
From: Johannes Schindelin <hidden> Date: 2016-06-15 22:42:17
Hi,
On Mon, 30 Jan 2006, Junio C Hamano wrote:
We need to realize that `upload-pack` that hears
"have A, want B" is allowed to omit objects that appear in
`ls-tree B` output but not in `ls-tree A`. "have A" means not
just "I have A", but "I have A and all of its ancestors", so
just sending "have start_shallow" (or start_shallow^ for that
matter) is not quite enough.
So how about adding a "have-single A" which would be translated to
"git-rev-list ~A", which in turn would only mark the tree and its
children, but not the parents?
Ciao,
Dscho