Tracking a repository for content instead of history

Hello,

For interests sake I'd like to track the kernel.org linux repository.  
However, I'm not that bothered about tracking the history - it's more that I 
like to have the latest kernel release lying around.

Is there a way that I could just pull individual commits from a git 
repository?  In particular - could I make a repository (obviously not a 
clone, because it wouldn't have all the history) that contained only the 
tagged commits from an upstream repository?

Is it even sensible to want that?  It strikes me that it's possible that there 
isn't that much space/bandwidth saving to be made.  Should I just clone the 
repository and shut up?  :-)


Andy
-- 
Dr Andy Parkins, M Eng (hons), MIEE

On Tuesday 2006 December 12 14:28, Johannes Schindelin wrote:

You are not by any chance talking about the --remote option to
git-archive?

I wasn't; but that's certainly a helpful switch.  It's certainly a huge help.

If you want to reduce the number of objects to be downloaded, by telling
the other side what you have, you literally end up with something like
shallow clone: the other side _has_ to support it.

I suppose so; but I was thinking more an automated way of getting the data 
that is supplied for the kernel anyway.  So:

base-v1.0.0.tar.gz
patch-v1.0.1.gz
patch-v1.0.2.gz
etc

Each patch is obviously smaller than "base".  Git could easily make the 
patches, and each of those patches could be fed by hand into a repository 
with git-apply.  It doesn't seem like something that would require support on 
the other side, because it isn't so much a shallow clone (which /would/ 
preserve history, making it available if wanted); it is pulling just, say, 
tagged commits out of an existing repository.

Given a list of tags it is almost:

git-archive <get me base>
ssh remote git-diff v1.0.0..v1.0.1 | git-apply; git commit
ssh remote git-diff v1.0.1..v1.0.2 | git-apply; git commit

If that makes sense?  Obviously though it would be possible to use git rather 
than ssh to do this.

However... please don't waste any more time thinking about this; it's not a 
problem I have that needs a solution - it was more a "because I'm curious" 
sort of question.



Andy
-- 
Dr Andy Parkins, M Eng (hons), MIEE

Andy Parkins wrote:

For interests sake I'd like to track the kernel.org linux repository.  
However, I'm not that bothered about tracking the history - it's more that I 
like to have the latest kernel release lying around.

Is there a way that I could just pull individual commits from a git 
repository?  In particular - could I make a repository (obviously not a 
clone, because it wouldn't have all the history) that contained only the 
tagged commits from an upstream repository?

As of beta (in 'next') you can do 'shallow clone'm i.e. clone/fetch
only N commits depth history.
 

Is it even sensible to want that?  It strikes me that it's possible that there 
isn't that much space/bandwidth saving to be made.  Should I just clone the 
repository and shut up?  :-)

I've had similar idea: search for "sparse clone" keyword. But no code.

-- 
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git

Hi,

On Tue, 12 Dec 2006, Andy Parkins wrote:

On Tuesday 2006 December 12 14:28, Johannes Schindelin wrote:

quoted

You are not by any chance talking about the --remote option to
git-archive?

I wasn't; but that's certainly a helpful switch.  It's certainly a huge 
help.

quoted

If you want to reduce the number of objects to be downloaded, by telling
the other side what you have, you literally end up with something like
shallow clone: the other side _has_ to support it.

I suppose so; but I was thinking more an automated way of getting the data 
that is supplied for the kernel anyway.  So:

base-v1.0.0.tar.gz
patch-v1.0.1.gz
patch-v1.0.2.gz
etc

Each patch is obviously smaller than "base".  Git could easily make the 
patches, and each of those patches could be fed by hand into a repository 
with git-apply.

If it weren't for the recent discussion of kernel.org being overloaded 
with gitweb processes, I'd just write down a hint like 
http://repo.or.cz/w/git/jnareb-git.git?a=commitdiff_plain;h=next;hp=master

But since kernel.org is overloaded, I will not do that.

Ciao,

Hi,

On Tue, 12 Dec 2006, Andy Parkins wrote:

The way I would do it given nothing else is to simply extract snapshots 
into a working directory; and create a repository from scratch.  I was 
just wondering if a method existed that could reduce the size of the 
download.

You are not by any chance talking about the --remote option to 
git-archive?

If you want to reduce the number of objects to be downloaded, by telling 
the other side what you have, you literally end up with something like 
shallow clone: the other side _has_ to support it.

Ciao,
Dscho

On 12/13/06, Nguyen Thai Ngoc Duy [off-list ref] wrote:

Hm.. I'm no git:// expert. But is it possible doing as follow?
1. git-archive <base>
2. reconstruct commit, blobs and trees from the archive
3. tell git server that you have one commit, you need another commit
(maybe heads only, i'm not sure here)
4. get the pack from git server, create new commit and a diff

Ok. Stupid idea. The pack may base on objects that I don't have.

-- 

Hi,

On Wed, 13 Dec 2006, Nguyen Thai Ngoc Duy wrote:

On 12/13/06, Nguyen Thai Ngoc Duy [off-list ref] wrote:

quoted

Hm.. I'm no git:// expert. But is it possible doing as follow?
1. git-archive <base>
2. reconstruct commit, blobs and trees from the archive
3. tell git server that you have one commit, you need another commit
(maybe heads only, i'm not sure here)
4. get the pack from git server, create new commit and a diff

Ok. Stupid idea. The pack may base on objects that I don't have.

The only not-so-brilliant idea is to reconstruct the commit from the 
archive. This is not possible, as not only some author and 
committer metadata is not reconstructable, but worse: the parents' hash is 
not either. And since all these are hashed to get the commit hash, you 
lost.

However, it could work like this:

- reconstruct tree commit
- ask for a diff between a certain commit, with respect to your tree

It might even be easy to convince git-upload-pack to construct a thin pack 
containing deltas _only_ against objects which are reachable from your 
tree.

Note: this is feasible, but not necessarily sensible:

- it puts more strain on the server, which otherwise could probably reuse 
a lot of deltas, and
- it contradicts the idea of _distributed_ development (for example, you 
could not tell which HEAD commit is newer when you fetched from two 
repos).

Probably, you could add a third argument: merges are not necessarily 
_possible_ with that setup. Note that this argument applies to shallow 
clones, too!

Ciao,
Dscho

On Tuesday 2006 December 12 13:04, Jakub Narebski wrote:

quoted

Is it even sensible to want that?  It strikes me that it's possible that
there isn't that much space/bandwidth saving to be made.  Should I just
clone the repository and shut up?  :-)

I've had similar idea: search for "sparse clone" keyword. But no code.

While the functionality might not be built into git in terms of clone, would 
there be a way to pull a particular commit from another repository? 

The way I would do it given nothing else is to simply extract snapshots into a 
working directory; and create a repository from scratch.  I was just 
wondering if a method existed that could reduce the size of the download.

I think the best way is going to be to use the patches published at kernel.org 
and apply them one at a time with git-apply.


Andy
-- 
Dr Andy Parkins, M Eng (hons), MIEE

On 12/12/06, Andy Parkins [off-list ref] wrote:

I suppose so; but I was thinking more an automated way of getting the data
that is supplied for the kernel anyway.  So:

base-v1.0.0.tar.gz
patch-v1.0.1.gz
patch-v1.0.2.gz
etc

Each patch is obviously smaller than "base".  Git could easily make the
patches, and each of those patches could be fed by hand into a repository
with git-apply.  It doesn't seem like something that would require support on
the other side, because it isn't so much a shallow clone (which /would/
preserve history, making it available if wanted); it is pulling just, say,
tagged commits out of an existing repository.

Given a list of tags it is almost:

git-archive <get me base>
ssh remote git-diff v1.0.0..v1.0.1 | git-apply; git commit
ssh remote git-diff v1.0.1..v1.0.2 | git-apply; git commit

If that makes sense?  Obviously though it would be possible to use git rather
than ssh to do this.

Hm.. I'm no git:// expert. But is it possible doing as follow?
1. git-archive <base>
2. reconstruct commit, blobs and trees from the archive
3. tell git server that you have one commit, you need another commit
(maybe heads only, i'm not sure here)
4. get the pack from git server, create new commit and a diff
-- 

Hi,

On Tue, 12 Dec 2006, Johannes Schindelin wrote:

If it weren't for the recent discussion of kernel.org being overloaded 
with gitweb processes, I'd just write down a hint like 
[URL edited out]

But since kernel.org is overloaded, I will not do that.

Side note: it would probably not help you. The diff is uncompressed, and 
thus likely _substantially larger_ than getting the snapshot via gitweb, 
which _is_ compressed.

Ciao,
Dscho