From: Petr Baudis <hidden> Date: 2016-06-15 22:42:01
Hello,
I'm happy to announce the release of the 0.12 version of the Cogito
SCM-like layer over Linus' GIT tree history storage tool. Get it at
http://www.kernel.org/pub/software/scm/cogito/
or cg-update if you have an older version cloned.
I wanted to release it later with more cool features, but after all
releasing often is good and people will get to test things more, and
I wanted to make it possible for kernel.org to upgrade to newer RPM.
But it may not be as stable as I'd wish and may have some rough edges,
so be warned.
This release contains the latest stuff from Linus, with all the
packing stuff and everything. Other things include heaps of bugfixes,
enhanced options parsing, ~/.cgrc support, cg-push, real cg-tag, and
plenty of smaller but nice stuff. And more to come in next days!
About cg-push, it:
(i) works only locally or over git+ssh branches
(ii) the head updated on the other side must be 'master' too
(high priority to fix)
(iii) the head updated on the other side is re-created, thus losing
all attributes (ownership, permissions)
(high priority to fix)
(iv) won't update the remote working tree if there is any associated
with the repository - do cg-cancel to catch up, but that will
lose any local changes you did (note that I plan to rename
cg-cancel to cg-reset)
Also, I've deprecated rsync, as I explained in another mail. Use
cg-branch-chg to change the branch URLs to some more sensible scheme -
most likely HTTP, or SSH if you want to push as well.
Have fun,
--
Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
<Espy> be careful, some twit might quote you out of context..
From: Brian Gerst <hidden> Date: 2016-06-15 22:42:01
Petr Baudis wrote:
Hello,
I'm happy to announce the release of the 0.12 version of the Cogito
SCM-like layer over Linus' GIT tree history storage tool. Get it at
http://www.kernel.org/pub/software/scm/cogito/
or cg-update if you have an older version cloned.
I wanted to release it later with more cool features, but after all
releasing often is good and people will get to test things more, and
I wanted to make it possible for kernel.org to upgrade to newer RPM.
But it may not be as stable as I'd wish and may have some rough edges,
so be warned.
This release contains the latest stuff from Linus, with all the
packing stuff and everything. Other things include heaps of bugfixes,
enhanced options parsing, ~/.cgrc support, cg-push, real cg-tag, and
plenty of smaller but nice stuff. And more to come in next days!
About cg-push, it:
(i) works only locally or over git+ssh branches
(ii) the head updated on the other side must be 'master' too
(high priority to fix)
(iii) the head updated on the other side is re-created, thus losing
all attributes (ownership, permissions)
(high priority to fix)
(iv) won't update the remote working tree if there is any associated
with the repository - do cg-cancel to catch up, but that will
lose any local changes you did (note that I plan to rename
cg-cancel to cg-reset)
Also, I've deprecated rsync, as I explained in another mail. Use
cg-branch-chg to change the branch URLs to some more sensible scheme -
most likely HTTP, or SSH if you want to push as well.
I really question removing rsync before HTTP pulls become more
effecient. I did a complete pull of cogito from kernel.org, and http
took over 50 minutes to pull everything, while rsync was done in just
over 1 minute. I dared not even try to pull the full kernel at that speed.
I suspect that part of the problem is that the pull methods are doing a
depth first search, so we can't request the next object until the
current object is fully received and parsed. Changing to a breadth
first search would allow multiple requests in flight and asynchronous
processing which should speed things up. I am exploring using the
curl_multi_* functions to do this, but this will require changes to
common code in pull.c.
--
Brian Gerst
From: Petr Baudis <hidden> Date: 2016-06-15 22:42:01
Dear diary, on Wed, Jul 06, 2005 at 02:01:38PM CEST, I got a letter
where Brian Gerst [off-list ref] told me that...
Petr Baudis wrote:
quoted
Also, I've deprecated rsync, as I explained in another mail. Use
cg-branch-chg to change the branch URLs to some more sensible scheme -
most likely HTTP, or SSH if you want to push as well.
I really question removing rsync before HTTP pulls become more
effecient.
It won't happen. Or rather, I hope the HTTP pulls become more efficient
soon. Actually, perhaps Linus has something done already, my workstation
is a bit derailed now so I couldn't pull from him in the last few days
(hopefully will sort that out today).
I did a complete pull of cogito from kernel.org, and http
took over 50 minutes to pull everything, while rsync was done in just
over 1 minute. I dared not even try to pull the full kernel at that speed.
I suspect that part of the problem is that the pull methods are doing a
depth first search, so we can't request the next object until the
current object is fully received and parsed. Changing to a breadth
first search would allow multiple requests in flight and asynchronous
processing which should speed things up. I am exploring using the
curl_multi_* functions to do this, but this will require changes to
common code in pull.c.
Hmm, yes, I guess Linus won't be touching the HTTP backend at all. ;-) I
suggest you to check the last development in Linus' branch and sync with
Daniel Barkalow, who promised improving the pull tools as well.
--
Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
<Espy> be careful, some twit might quote you out of context..
From: Junio C Hamano <hidden> Date: 2016-06-15 22:42:01
quoted
quoted
quoted
quoted
"PB" == Petr Baudis [off-list ref] writes:
PB> It won't happen. Or rather, I hope the HTTP pulls become more efficient
PB> soon. Actually, perhaps Linus has something done already, my workstation
PB> is a bit derailed now so I couldn't pull from him in the last few days
PB> (hopefully will sort that out today).
PB> Hmm, yes, I guess Linus won't be touching the HTTP backend at all. ;-) I
PB> suggest you to check the last development in Linus' branch and sync with
PB> Daniel Barkalow, who promised improving the pull tools as well.
If this weekend is not too late, I have been brewing what is
called an "efficient pull from dumb servers" suite, which would
hopefully fill this gap. I am still in the process of finishing
the details, but basically it already seems to work.
Linus, please drop the patch I sent you earlier, privately by
mistake not CCing the list, that implemented only the server
end. I've changed some file formats already from that one.
The outline of how it works is like this.
* I assume a dumb transport (read: static files only HTTP
server) and no on-request server side processing. All the
smarts must go in the client. The server side X.git being an
ordinary GIT archive (no need for files in the work tree),
plus:
- X.git/objects/pack can have packed GIT archives. I
envision that this will be a series of 5 to 20 MB packs,
occasionally adding a new incremental pack when
X.git/objects/??/ directories accumulate enough standalone
SHA1 files. It is not necessary to have X.git/objects/??/
files if an object is contained in one of the packs.
- X.git/info/ has three extra files.
- "inventory" lists all the branches stored in X.git/refs
and looks like this (contents and path):
ff83c8f3554ceb444b413beaeb49b4a781dae944 snap/0
013e7c7ff498aae82d799f80da37fbd395545456 snap/10
ff83c8f3554ceb444b413beaeb49b4a781dae944 heads/master
dd7ba8b4949535c24e604a37709db0e3be9ccbbc heads/linus
This is to facilitate discovery from a transport that is
not so "ls" friendly, like HTTP.
- "pack" lists available packs under X.git/objects/pack and
looks like this (size and name):
432495 pk-65fe69e9bc2e8a3e0881e008dde182522156ba7c.pack
The file is there for discovery. The size is used by the
client to discover optimum set of packs to slurp.
- "rev-cache" is a binary file that describes commit
ancestry information in a dense format. It lists all
commits available from this repository along with who
its parents are for each of the commit. This file is
produced append-only, so that the server side can use
rsync based mirroring scheme.
A new command "git-update-dumb-server" is used to prepare
these three files. There may need a helper script that uses
git-pack-objects and friends to prepare packs partitioned to
allow pulling a popular branch efficiently.
* The client side is called "git-dumb-pull-script". This
downloads the above three files, and .idx files associated
with packs described in "pack". With the information in
"inventory" about desired branch to pull from along with
"rev-cache" ancestry information, it discovers the set of
commits that is lacking from its local store. By comparing
that list with downloaded .idx files, along with size
information for each pack, it comes up a list of packs to
download to cover the most commits that it wants to obtain,
and downloads them, verifies them and stores them in its
.git/objects/pack/ directory.
The above process of downloading packs would typically not
cover all the things lacking, because some new commits may
not be in any of the packs. After this point, the usual
commit-walking git-http-pull can be used to fill the rest,
and it does not have to pull that many objects. Dan's
http-pull parallelism improvement would be very useful
independently here.
- X.git/objects/pack can have packed GIT archives. I
envision that this will be a series of 5 to 20 MB packs,
occasionally adding a new incremental pack when
X.git/objects/??/ directories accumulate enough standalone
SHA1 files. It is not necessary to have X.git/objects/??/
files if an object is contained in one of the packs.
Note that I just re-packed the kernel archive on kernel.org, and removed
_all_ unpacked files. Once that percolates to the mirrors, the http
protocol will be useless without anything like this.
That said, I really think the dumb protocols are useless anyway. No other
system supports pure static object pulling anyway, and as far as I'm
concerned, I want "rsync" to kind of work (but it won't be optimal, since
re-packing will delete all the old objects and replace it with the new
pack that is downloaded anew). But plain http? I'm not convinced.
I'd much rather have a "stupid server" that just listens to a port, and
basically forks off and executes "git-upload-pack" when it's connected to
(perhaps reading the directory name first). Nothing else. Then we can do
a security analysis of upload-pack, which should be fairly easy since it's
not actually ever _writing_ anything.
At that point, you can do
git pull git://www.kernel.org/pub/scm/git/..
and it would just connect to some default "git port", pass off the
directory name, and be done with it - exact same discovery protocol that
now use for ssh. And "git clone" would also automatically work.
Linus
From: Petr Baudis <hidden> Date: 2016-06-15 22:42:01
Let me join the sceptics camp. :-)
Dear diary, on Thu, Jul 07, 2005 at 09:04:58PM CEST, I got a letter
where Linus Torvalds [off-list ref] told me that...
Note that I just re-packed the kernel archive on kernel.org, and removed
_all_ unpacked files. Once that percolates to the mirrors, the http
protocol will be useless without anything like this.
*grumble*
So, what _is_ then the way to pull now, actually? If we use rsync, won't
we end up with having the objects we previous had twice now?
That said, I really think the dumb protocols are useless anyway. No other
system supports pure static object pulling anyway, and as far as I'm
concerned, I want "rsync" to kind of work (but it won't be optimal, since
re-packing will delete all the old objects and replace it with the new
pack that is downloaded anew). But plain http? I'm not convinced.
You can always just spider the repository which will work just as well
as rsync in the git case. ;-)
I think it would be actually simplest (for the user) to have a trivial
CGI script on the other side which will do the git-upload-pack stuff.
Minimal extra administrative overhead, flexibility, works through
proxies, and stuff. People can rewrite it in Perl or PHorridP if they
wish and use it on webhosting servers not allowing much else.
That's not to say a dedicated server wouldn't have its place too, and
that's what's now probably simplest for us. ;-)
Now we are in a situation when there's actually no way to pull from your
kernel repository without throwing own repository to mess and
duplicating data, AFAICS.
I'd much rather have a "stupid server" that just listens to a port, and
basically forks off and executes "git-upload-pack" when it's connected to
(perhaps reading the directory name first). Nothing else. Then we can do
a security analysis of upload-pack, which should be fairly easy since it's
not actually ever _writing_ anything.
At that point, you can do
git pull git://www.kernel.org/pub/scm/git/..
and it would just connect to some default "git port", pass off the
directory name, and be done with it - exact same discovery protocol that
now use for ssh. And "git clone" would also automatically work.
Eek. Could you please make it at least pretend to be extensible? Compare
git-upload-pack with git-ssh-pu* - the second one prepends letters to
the data it sends so that if you add a new type of stuff to send (say
for authentication or some smart tags stuff), you could extend it in a
sensible way. What about dividing the communication to "blocks"
separated by a newline? Each block would have its first word on the
first line saying what kind of block it is - "refs", "have", "want", or
"pack" (for simplicity, the pack block might have additional restriction
that it's always the last one). If you hit unknown block, you should
respond back by something like "huh" and ignore the rest of it.
--
Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
<Espy> be careful, some twit might quote you out of context..
Let me join the sceptics camp. :-)
Dear diary, on Thu, Jul 07, 2005 at 09:04:58PM CEST, I got a letter
where Linus Torvalds [off-list ref] told me that...
quoted
Note that I just re-packed the kernel archive on kernel.org, and removed
_all_ unpacked files. Once that percolates to the mirrors, the http
protocol will be useless without anything like this.
*grumble*
So, what _is_ then the way to pull now, actually? If we use rsync, won't
we end up with having the objects we previous had twice now?
Rsync works fine. You can either unpack the pack you get, or, if you
prefer, just run
git-prune-packed
which will remove the stand-alone object that it finds in packs. Now
you're no longer duplicating data, and your repository is smaller than it
used to be anyway.
Of course, that requires that you trust the packs 100%. It seems to be
stable, and I've packed the whole kernel repo, but I actually keep my
private tree unpacked still just in case.
I think it would be actually simplest (for the user) to have a trivial
CGI script on the other side which will do the git-upload-pack stuff.
Well, git-upload-pack expects the other end to follow the proper protocol,
but yes, you can certainly expose it through a web interface and a
specialized client that way.
Linus
From: Tony Luck <hidden> Date: 2016-06-15 22:42:01
quoted
So, what _is_ then the way to pull now, actually? If we use rsync, won't
we end up with having the objects we previous had twice now?
Rsync works fine. You can either unpack the pack you get, or, if you
prefer, just run
git-prune-packed
cg-update from a local repo that contains packs is broken though :-(
Also "git-fsck-cache" in a repo that is fully packed complains:
fatal: No default references
-Tony
From: Tony Luck <hidden> Date: 2016-06-15 22:42:01
quoted
cg-update from a local repo that contains packs is broken though :-(
Is this with cg-0.12? The most recent release should be happy with packs.
Yes ... I pulled, built and installed the latest cogito this afternoon
before trying
to touch anything involving packs. cg-version says:
cogito-0.12 (b21855b8734ca76ea08c0c17e4a204191b6e3add)
This is what happens ("linus" is a local branch just pulled from kernel.org,
so it just contains one pack file and its index).
$ cg-update linus
`/home/aegl/GIT/linus/.git/refs/heads/master' -> `.git/refs/heads/linus'
does not exist /home/aegl/GIT/linus/.git/objects/04/3d051615aa5da09a7e44f1edbb69
798458e067
Cannot obtain needed object 043d051615aa5da09a7e44f1edbb69798458e067
while processing commit 0000000000000000000000000000000000000000.
cg-pull: objects pull failed
If I try it again, it thinks things are up to date (since it
mistakenly updated the
.git/refs/heads/linus), but then fails to apply (since it doesn't have
the objects
it needs).
-Tony
cg-update from a local repo that contains packs is broken though :-(
Is this with cg-0.12? The most recent release should be happy with packs.
Ahh, I see it. It's because it uses "git-local-pull", and yes,
git-local-pull does the old filename assumption. Right?
Ho humm.. That's a bug in local-pull.c, although I'm not sure how to fix
it best. One option is to just not use it (as in "use git-fetch-pack
instead"), and another is to use GIT_ALTERNATE_OBJECT_DIRECTORIES and just
pick up the files that way. Yet another one is to actually copy over (or
link) the pack-file, but that's likely the least preferable one.
The _simplest_ fix is to use git-fetch-pack. It doesn't give you the
convenient hard-linking, though.
Linus
This is what happens ("linus" is a local branch just pulled from kernel.org,
so it just contains one pack file and its index).
$ cg-update linus
`/home/aegl/GIT/linus/.git/refs/heads/master' -> `.git/refs/heads/linus'
does not exist /home/aegl/GIT/linus/.git/objects/04/3d051615aa5da09a7e44f1edbb69
798458e067
Cannot obtain needed object 043d051615aa5da09a7e44f1edbb69798458e067
while processing commit 0000000000000000000000000000000000000000.
cg-pull: objects pull failed
Ok. The immediate fix is to just unpack the pack:
mv .git/objects/pack/* .git/
for i in .git/*.pack; do git-unpack-objects < $i; done
(or similar - the above is untested, but I think it should be obvious
enough what I'm trying to do).
If I try it again, it thinks things are up to date (since it mistakenly
updated the .git/refs/heads/linus), but then fails to apply (since it
doesn't have the objects it needs).
Ok, that's a worse bug, it really shouldn't update the head until _after_
the pull has succeeded.
Linus
From: Petr Baudis <hidden> Date: 2016-06-15 22:42:01
Dear diary, on Fri, Jul 08, 2005 at 02:09:48AM CEST, I got a letter
where Linus Torvalds [off-list ref] told me that...
On Thu, 7 Jul 2005, Linus Torvalds wrote:
quoted
quoted
cg-update from a local repo that contains packs is broken though :-(
Is this with cg-0.12? The most recent release should be happy with packs.
Ahh, I see it. It's because it uses "git-local-pull", and yes,
git-local-pull does the old filename assumption. Right?
Ho humm.. That's a bug in local-pull.c, although I'm not sure how to fix
it best.
It seems like the whole pull family is totally borked now, and I'm
getting desperate. Looks like this evening will be *pull.c fixing for
me.
Jul 04 Daniel Barkalow [PATCH 0/2] Support for transferring pack files in git-ssh-*
is what brings some hope to my life, though. Daniel? Any chance we could
get the similar fixes for local-pull? (I didn't actually look at the
patch but briefly.) I'll try to review the ssh patchset ASAP - I still
prefer it much to the fetch-pack things since its protocol is actually
extensible.
The _simplest_ fix is to use git-fetch-pack. It doesn't give you the
convenient hard-linking, though.
Hard-linking is an absolute must for local repositories (well, either
that for people who want safety, or symlinking for the rest who want
speed - I want to make that one possible in Cogito ASAP but it requires
some non-trivial changes to some of its assumptions).
--
Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
<Espy> be careful, some twit might quote you out of context..
From: Daniel Barkalow <hidden> Date: 2016-06-15 22:42:01
On Fri, 8 Jul 2005, Petr Baudis wrote:
It seems like the whole pull family is totally borked now, and I'm
getting desperate. Looks like this evening will be *pull.c fixing for
me.
Jul 04 Daniel Barkalow [PATCH 0/2] Support for transferring pack files in git-ssh-*
is what brings some hope to my life, though. Daniel? Any chance we could
get the similar fixes for local-pull? (I didn't actually look at the
patch but briefly.)
This patch is not actually for transferring objects which are in pack
files in the source, but for transferring a group of objects as a pack
file. It does, however, read the source side with git-pack-objects to
generate the content to send, so it would, I guess, fix the problem for
the case where it decides to use a pack to transfer.
The real fix is to go through the pull methods (local-pull and
ssh-pull; http-pull presumably won't be encountering pack files yet) and
make them do appropriate things with pack files.
One thing that is in the patch is a change to the comment, specifying
that fetch() could also get other objects in addition to the one
specified, if there's some reason to think this is a good idea; the fix
for local-pull is probably to link/symlink/copy the pack file if the
object is in one.
For ssh-pull, serve_object in ssh-push needs to be taught how to extract
an object from a pack file and send it.
However, there's a bug in pull.c, covering up a terrible performance
issue: it doesn't actually make sure you have all the parent of a commit
that you had when it checked (due to not having a way of caching the
result of checking this, which would require you to put the entire
repository through cache each time you pull). This would mean that, if you
have a pack that references something outside of it, you won't get
everything with my proposal above.
I should be able to spend some time on these issues over the weekend.
-Daniel
*This .sig left intentionally blank*