From: Russell King <hidden> Date: 2016-06-15 22:41:55
Ok. cogito-0.8.
rmk@dyn-67:[linux-2.6-rmk]:<1049> cg-update origin
`../linux-2.6/.git/refs/heads/master' -> `.git/refs/heads/origin'
`../linux-2.6/.git/objects/00/78aeb85737197a84af1eeb0353dbef74427901' -> `.git/objects/00/78aeb85737197a84af1eeb0353dbef74427901'
cp: cannot create link `.git/objects/00/78aeb85737197a84af1eeb0353dbef74427901': File exists
`../linux-2.6/.git/objects/01/ca31cc7bfdd18d6f72288915021730442f386d' -> `.git/objects/01/ca31cc7bfdd18d6f72288915021730442f386d'
cp: cannot create link `.git/objects/01/ca31cc7bfdd18d6f72288915021730442f386d': File exists
`../linux-2.6/.git/objects/04/17820d15efac837a84f9ade46f56339016a282' -> `.git/objects/04/17820d15efac837a84f9ade46f56339016a282'
cp: cannot create link `.git/objects/04/17820d15efac837a84f9ade46f56339016a282': File exists
`../linux-2.6/.git/objects/07/da010b67d4d715b5e97dfb824eb70433776a20' -> `.git/objects/07/da010b67d4d715b5e97dfb824eb70433776a20'
cp: cannot create link `.git/objects/07/da010b67d4d715b5e97dfb824eb70433776a20': File exists
`../linux-2.6/.git/objects/07/5d3961a119e8f27294cd77193f8fee7908a521' -> `.git/objects/07/5d3961a119e8f27294cd77193f8fee7908a521'
...
cp: cannot create link `.git/objects/fc/1428905472a61e8e51057a4237acab5d8594d8': File exists
`../linux-2.6/.git/objects/fc/373c483e62dc1bbc5c3915f2d3c795fb316ec5' -> `.git/objects/fc/373c483e62dc1bbc5c3915f2d3c795fb316ec5'
cp: cannot create link `.git/objects/fc/373c483e62dc1bbc5c3915f2d3c795fb316ec5': File exists
`../linux-2.6/.git/objects/ff/c3be3dff7e20e2ad5367fa8d6d0d2f0baa8a24' -> `.git/objects/ff/c3be3dff7e20e2ad5367fa8d6d0d2f0baa8a24'
cp: cannot create link `.git/objects/ff/c3be3dff7e20e2ad5367fa8d6d0d2f0baa8a24': File exists
`../linux-2.6/.git/objects/ff/8b49966a9f6ed23f6489bb986de87a14d4b783' -> `.git/objects/ff/8b49966a9f6ed23f6489bb986de87a14d4b783'
cp: cannot create link `.git/objects/ff/8b49966a9f6ed23f6489bb986de87a14d4b783': File exists
cg-pull: rsync error
rmk@dyn-67:[linux-2.6-rmk]:<1052> md5sum .git/objects/ff/8b49966a9f6ed23f6489bb986de87a14d4b783 ../linux-2.6/.git/objects/ff/8b49966a9f6ed23f6489bb986de87a14d4b783
194b70d0eed786807e14e97dd0a5ad8d .git/objects/ff/8b49966a9f6ed23f6489bb986de87a14d4b783
194b70d0eed786807e14e97dd0a5ad8d ../linux-2.6/.git/objects/ff/8b49966a9f6ed23f6489bb986de87a14d4b783
rmk@dyn-67:[linux-2.6-rmk]:<1053> vdir .git/objects/ff/8b49966a9f6ed23f6489bb986de87a14d4b783 ../linux-2.6/.git/objects/ff/8b49966a9f6ed23f6489bb986de87a14d4b783
-rw-rw-r-- 1 rmk rmk 3070 Apr 28 10:43 .git/objects/ff/8b49966a9f6ed23f6489bb986de87a14d4b783
-rw-r--r-- 1 rmk rmk 3070 Apr 29 16:50 ../linux-2.6/.git/objects/ff/8b49966a9f6ed23f6489bb986de87a14d4b783
What seems to be happening is that I create changes in my tree, rsync
them out to kernel.org. Linus pulls them into his tree. I pull them
back into my reference tree, and then try and update my working tree.
By that time, the object files in the reference tree appear to have
a newer timestamp than the corresponding ones in my local tree, and
cp -lua fails.
Which means cogito fails to work for me... Help.
--
Russell King
From: Petr Baudis <hidden> Date: 2016-06-15 22:41:55
Dear diary, on Fri, Apr 29, 2005 at 06:01:27PM CEST, I got a letter
where Russell King [off-list ref] told me that...
rmk@dyn-67:[linux-2.6-rmk]:<1049> cg-update origin
`../linux-2.6/.git/objects/00/78aeb85737197a84af1eeb0353dbef74427901' -> `.git/objects/00/78aeb85737197a84af1eeb0353dbef74427901'
cp: cannot create link `.git/objects/00/78aeb85737197a84af1eeb0353dbef74427901': File exists
By that time, the object files in the reference tree appear to have
a newer timestamp than the corresponding ones in my local tree, and
cp -lua fails.
I'm now away ,unfortunately, and no immediate idea stems to my mind on
how to fix it. Ideas welcomed - I need to hardlink missing entries from
one tree to another; it would be enough to be able to just tell cp to
ignore already present files.
Could you please try to give cp the -f flag?
--
Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
C++: an octopus made by nailing extra legs onto a dog. -- Steve Taylor
From: Ryan Anderson <hidden> Date: 2016-06-15 22:41:55
On Fri, Apr 29, 2005 at 08:27:08PM +0200, Petr Baudis wrote:
Dear diary, on Fri, Apr 29, 2005 at 06:01:27PM CEST, I got a letter
where Russell King [off-list ref] told me that...
quoted
rmk@dyn-67:[linux-2.6-rmk]:<1049> cg-update origin
`../linux-2.6/.git/objects/00/78aeb85737197a84af1eeb0353dbef74427901' -> `.git/objects/00/78aeb85737197a84af1eeb0353dbef74427901'
cp: cannot create link `.git/objects/00/78aeb85737197a84af1eeb0353dbef74427901': File exists
By that time, the object files in the reference tree appear to have
a newer timestamp than the corresponding ones in my local tree, and
cp -lua fails.
I'm now away ,unfortunately, and no immediate idea stems to my mind on
how to fix it. Ideas welcomed - I need to hardlink missing entries from
one tree to another; it would be enough to be able to just tell cp to
ignore already present files.
Could you please try to give cp the -f flag?
Why not just use "rsync" for both remote and local synchronization, and
provide a "relink" command to scan two .git/objects/ repositories and
hardlink matching files together?
With the SHA1 hash, you can even have a --unsafe option that just
compares the has names and does a link based purely off of that and the
stat(2) results of both files. (I'd expect that a ... safer variant
would extract both files and compare them, but the --unsafe should be
sufficient, in practice, I would think.)
--
Ryan Anderson
sometimes Pug Majere
From: Thomas Glanzmann <hidden> Date: 2016-06-15 22:41:55
Hello,
Why not just use "rsync" for both remote and local synchronization, and
provide a "relink" command to scan two .git/objects/ repositories and
hardlink matching files together?
That came to my mind, too. And it is actually the only thing that makes
sense. - In matters of KISS. :-)
Thomas
Why not just use "rsync" for both remote and local synchronization, and
provide a "relink" command to scan two .git/objects/ repositories and
hardlink matching files together?
Absolutely. I use the same "git-pull-script" between two local directories
on disk. The only issue there is that you have to give the ".git"
directory, ie you should do
git-pull-script ~/by/other/repository/.git
instead of pointing to the other repo's root.
Of course, I don't bother with the linking. But that's the trivial part.
With the SHA1 hash, you can even have a --unsafe option that just
compares the has names and does a link based purely off of that and the
stat(2) results of both files. (I'd expect that a ... safer variant
would extract both files and compare them, but the --unsafe should be
sufficient, in practice, I would think.)
I don't think there is any point to unsafe. The assumption is that if you
do things this way, the "unlinked" files will the the uncommon case, so
what you do is
- remember the list of files you copied when you did the pull (you had to
have this list at some point anyway). Sort by name,
- create a list of names of both repositories, sorted by name
- do the union of those three lists (cheap, thanks to the sorting)
- stat each name to see if it's already linked (which it will be, most of
the time), continue to the next one..
- if they aren't linked, just do a "cmp" on them, and warn if they aren't
the same, continue to the next one.
- else link them.
And if you want to, you can skip the first stage, and just relink two
trees without looking at a list of "known new" files - it's going to be
expensive to link two big repositories the _first_ time, but hey even the
"expensive" part is likely to be pretty cheap in the end. If it takes an
hour or two to relink some years of history, big deal. Do it overnight,
you only need it once.
Linus
From: Petr Baudis <hidden> Date: 2016-06-15 22:41:55
Dear diary, on Fri, Apr 29, 2005 at 09:50:55PM CEST, I got a letter
where Ryan Anderson [off-list ref] told me that...
Why not just use "rsync" for both remote and local synchronization, and
provide a "relink" command to scan two .git/objects/ repositories and
hardlink matching files together?
No. This completely misses the point, which is to avoid useless I/O when
doing this local stuff; also, it saves disk space to a degree, but it is
wildly fluctuating.
I like Junio's local-pull solution much more (from the conceptual
standpoint; I didn't look at the code yet).
--
Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
C++: an octopus made by nailing extra legs onto a dog. -- Steve Taylor
From: Ryan Anderson <hidden> Date: 2016-06-15 22:41:56
Add git-relink-script, which will find common objects in two git
repositories and replace one copy with a hardlink.
Signed-Off-By: Ryan Anderson <redacted>
---
commit a3bcc763d71bdb91a3b48e9105fbaa5e79abb807
tree 2553e2d8befbe0cda3e413616fd4cc7bf04157ad
parent a31c6d022e2435a514fcc8ca57f9995c4376a986
author Ryan Anderson <ryan@mythryan2.(none)> 1115185675 -0400
committer Ryan Anderson [off-list ref] 1115185675 -0400
Index: Makefile
===================================================================
@@ -0,0 +1,115 @@+#!/usr/bin/env perl+# Copyright 2005, Ryan Anderson <ryan@michonline.com>+# Distribution permitted under the GPL v2, as distributed+# by the Free Software Foundation.+# Later versions of the GPL at the discretion of Linus Torvalds+#+# Scan two git object-trees, and hardlink any common objects between them.++use 5.006;+use strict;+use warnings;++sub get_canonical_form($);+sub do_scan_directory($$$);+sub compare_two_files($$);++# stats+my $linked = 0;+my $already = 0;++my ($dir1, $dir2) = @ARGV;++if (!defined $dir1 || !defined $dir2) {+ print("Usage: $0 <dir1> <dir2>\nBoth dir1 and dir2 should contain a .git/objects/ subdirectory.\n");+ exit(1);+}++$dir1 = get_canonical_form($dir1);+$dir2 = get_canonical_form($dir2);++printf("Searching '%s' and '%s' for common objects and hardlinking them...\n",$dir1,$dir2);++opendir(D,$dir1 . "objects/")+ or die "Failed to open $dir1/objects/ : $!";++my @hashdirs = grep !/^\.{1,2}$/, readdir(D);+foreach my $hashdir (@hashdirs) {+ do_scan_directory($dir1, $hashdir, $dir2);+}++printf("Linked %d files, %d were already linked.\n",$linked, $already);+++sub do_scan_directory($$$) {+ my ($srcdir, $subdir, $dstdir) = @_;++ my $sfulldir = sprintf("%sobjects/%s/",$srcdir,$subdir);+ my $dfulldir = sprintf("%sobjects/%s/",$dstdir,$subdir);++ opendir(S,$sfulldir)+ or die "Failed to opendir $sfulldir: $!";++ foreach my $file (grep(!/\.{1,2}$/, readdir(S))) {+ my $sfilename = $sfulldir . $file;+ my $dfilename = $dfulldir . $file;++ compare_two_files($sfilename,$dfilename);++ }+ closedir(S);+}++sub compare_two_files($$) {+ my ($sfilename, $dfilename) = @_;++ # Perl's stat returns relevant information as follows:+ # 0 = dev number+ # 1 = inode number+ # 7 = size+ my @sstatinfo = stat($sfilename);+ my @dstatinfo = stat($dfilename);++ if (@sstatinfo == 0 && @dstatinfo == 0) {+ die sprintf("Stat of both %s and %s failed: %s\n",$sfilename, $dfilename, $!);++ } elsif (@dstatinfo == 0) {+ return;+ }++ if ( ($sstatinfo[0] == $dstatinfo[0]) &&+ ($sstatinfo[1] != $dstatinfo[1])) {+ if ($sstatinfo[7] == $dstatinfo[7]) {+ unlink($dfilename)+ or die "Unlink of $dfilename failed: $!\n";++ link($sfilename,$dfilename)+ or die "Failed to link $sfilename to $dfilename: $!\n" .+ "Git Repository containing $dfilename is probably corrupted, please copy '$sfilename' to '$dfilename' to fix.\n";++ $linked++;++ } else {+ die sprintf("ERROR: File sizes are not the same, cannot relink %s to %s.\n",+ $sfilename, $dfilename);+ }++ } elsif ( ($sstatinfo[0] == $dstatinfo[0]) &&+ ($sstatinfo[1] == $dstatinfo[1])) {+ $already++;+ }+}++sub get_canonical_form($) {+ my $dir = shift;+ my $original = $dir;++ die "$dir is not a directory." unless -d $dir;++ $dir .= "/" unless $dir =~ m#/$#;+ $dir .= ".git/" unless $dir =~ m#\.git/$#;++ die "$original does not have a .git/ subdirectory.\n" unless -d $dir;++ return $dir;+}