From: Ondrej Certik <hidden> Date: 2016-06-15 22:45:41
Hi,
I would like to export our whole git repository to patches, and then
reconstruct it again from scratch. Following the man page of "git
fast-export":
$ git clone git://git.sympy.org/sympy-full-history-20081023.git
$ cd sympy-full-history-20081023
$ git fast-export --all --export-marks=marks > patches
$ cd ..
$ mkdir sympy-new
$ cd sympy-new
$ git init
$ git fast-import --export-marks=marks < ../sympy-full-history-20081023/patches
git-fast-import statistics:
---------------------------------------------------------------------
Alloc'd objects: 25000
Total objects: 21355 ( 144 duplicates )
blobs : 8009 ( 0 duplicates 4529 deltas)
trees : 10627 ( 144 duplicates 9189 deltas)
commits: 2719 ( 0 duplicates 0 deltas)
tags : 0 ( 0 duplicates 0 deltas)
Total branches: 21 ( 26 loads )
marks: 1048576 ( 10728 unique )
atoms: 726
Memory total: 2880 KiB
pools: 2098 KiB
objects: 781 KiB
---------------------------------------------------------------------
pack_report: getpagesize() = 4096
pack_report: core.packedGitWindowSize = 33554432
pack_report: core.packedGitLimit = 268435456
pack_report: pack_used_ctr = 40706
pack_report: pack_mmap_calls = 2791
pack_report: pack_open_windows = 1 / 2
pack_report: pack_mapped = 26177739 / 35513414
---------------------------------------------------------------------
However, the repository is very different to the original one. It
contains only 191 patches:
$ git log --pretty=oneline | wc -l
191
and it only contains couple files. Compare this with the original repository:
$ git log --pretty=oneline | wc -l
2719
What am I doing wrong? Is there some other way to do it? I also tried
"git format-patch" and "git am" and that almost works, only it changes
hashes. Is there some way to tell "git am" to preserve the hash?
Thanks,
Ondrej
From: Michael J Gruber <hidden> Date: 2016-06-15 22:45:41
Ondrej Certik venit, vidit, dixit 25.11.2008 17:44:
Hi,
I would like to export our whole git repository to patches, and then
reconstruct it again from scratch. Following the man page of "git
fast-export":
$ git clone git://git.sympy.org/sympy-full-history-20081023.git
$ cd sympy-full-history-20081023
$ git fast-export --all --export-marks=marks > patches
$ cd ..
$ mkdir sympy-new
$ cd sympy-new
$ git init
$ git fast-import --export-marks=marks < ../sympy-full-history-20081023/patches
git-fast-import statistics:
---------------------------------------------------------------------
Alloc'd objects: 25000
Total objects: 21355 ( 144 duplicates )
blobs : 8009 ( 0 duplicates 4529 deltas)
trees : 10627 ( 144 duplicates 9189 deltas)
commits: 2719 ( 0 duplicates 0 deltas)
tags : 0 ( 0 duplicates 0 deltas)
Total branches: 21 ( 26 loads )
marks: 1048576 ( 10728 unique )
atoms: 726
Memory total: 2880 KiB
pools: 2098 KiB
objects: 781 KiB
---------------------------------------------------------------------
pack_report: getpagesize() = 4096
pack_report: core.packedGitWindowSize = 33554432
pack_report: core.packedGitLimit = 268435456
pack_report: pack_used_ctr = 40706
pack_report: pack_mmap_calls = 2791
pack_report: pack_open_windows = 1 / 2
pack_report: pack_mapped = 26177739 / 35513414
---------------------------------------------------------------------
However, the repository is very different to the original one. It
contains only 191 patches:
$ git log --pretty=oneline | wc -l
191
and it only contains couple files. Compare this with the original repository:
$ git log --pretty=oneline | wc -l
2719
I get the same stats (with the dups) but a perfect rev count, when I use
git log --all. The reason is that the history in the imported repo is
disconnected at various places (at tagging commits)! Your command counts
only the revs backwards to the first "disconnection".
So, the real issue is: Why has the result these cuts in the history?
I don't know, I just noticed that turning on rename and copy detection
makes git-fast-import crash, which shouldn't happen either. Something's
not right here. CC'ing the authors of im- and export.
BTW: Maybe you can accomplish what you want with different means? Why
export|import directly to git?
Michael
--
git 1.6.0.4.608.ga9645
From: Peter Baumann <hidden> Date: 2016-06-15 22:45:41
On Tue, Nov 25, 2008 at 05:44:28PM +0100, Ondrej Certik wrote:
Hi,
I would like to export our whole git repository to patches, and then
reconstruct it again from scratch. Following the man page of "git
fast-export":
Perhabs you are looking for git filter-branch, because it seems you want
to change the history in some way (e.g. remove a wrongly committed file)?
Nevertheless, I expect your shown commands below to procude the same
repo again, so you might be on something ...
-Peter
$ git clone git://git.sympy.org/sympy-full-history-20081023.git
$ cd sympy-full-history-20081023
$ git fast-export --all --export-marks=marks > patches
$ cd ..
$ mkdir sympy-new
$ cd sympy-new
$ git init
$ git fast-import --export-marks=marks < ../sympy-full-history-20081023/patches
git-fast-import statistics:
---------------------------------------------------------------------
Alloc'd objects: 25000
Total objects: 21355 ( 144 duplicates )
blobs : 8009 ( 0 duplicates 4529 deltas)
trees : 10627 ( 144 duplicates 9189 deltas)
commits: 2719 ( 0 duplicates 0 deltas)
tags : 0 ( 0 duplicates 0 deltas)
Total branches: 21 ( 26 loads )
marks: 1048576 ( 10728 unique )
atoms: 726
Memory total: 2880 KiB
pools: 2098 KiB
objects: 781 KiB
---------------------------------------------------------------------
pack_report: getpagesize() = 4096
pack_report: core.packedGitWindowSize = 33554432
pack_report: core.packedGitLimit = 268435456
pack_report: pack_used_ctr = 40706
pack_report: pack_mmap_calls = 2791
pack_report: pack_open_windows = 1 / 2
pack_report: pack_mapped = 26177739 / 35513414
---------------------------------------------------------------------
However, the repository is very different to the original one. It
contains only 191 patches:
$ git log --pretty=oneline | wc -l
191
and it only contains couple files. Compare this with the original repository:
$ git log --pretty=oneline | wc -l
2719
What am I doing wrong? Is there some other way to do it? I also tried
"git format-patch" and "git am" and that almost works, only it changes
hashes. Is there some way to tell "git am" to preserve the hash?
Thanks,
Ondrej
On Tue, Nov 25, 2008 at 06:31:41PM +0100, Michael J Gruber [off-list ref] wrote:
I don't know, I just noticed that turning on rename and copy detection
makes git-fast-import crash, which shouldn't happen either. Something's
not right here. CC'ing the authors of im- and export.
Could you please write a testcase that reproduces your problem?
From: Ondrej Certik <hidden> Date: 2016-06-15 22:45:41
On Tue, Nov 25, 2008 at 9:41 PM, Miklos Vajna [off-list ref] wrote:
On Tue, Nov 25, 2008 at 06:31:41PM +0100, Michael J Gruber [off-list ref] wrote:
quoted
I don't know, I just noticed that turning on rename and copy detection
makes git-fast-import crash, which shouldn't happen either. Something's
not right here. CC'ing the authors of im- and export.
Could you please write a testcase that reproduces your problem?
quoted
Why export|import directly to git?
I guess he did not know about filter-branch. :)
I know about filter-branch (but I am not sure it can do what I want).
I made a mistake of not explaining what I want, instead I suggested (a
possibly wrong) solution. I want to export the whole git repository as
a set of human readable patches, that can be assembled back into a git
repository (with the same hashes as the original one) if needed. The
reason I want that is that if we later decide to switch to another
VCS, we have all the information to reproduce the repository. Another
reason is to be sure that we know all the sources that are needed to
construct the repository, e.g. that there are no binary blobs
(possibly containing malicious code). Another reason I want that is to
be able to rewrite the history, in particular, we have one Mercurial
repository with some old history and another Mercurial history with a
newer history and I just want to concatenate them together into one
git repository.
In each case I know several workarounds, but if there is a way to just
convert the whole git repository into a set of patches and (and be
able to convert everything back including the same hashes), then it'd
be awesome.
See also this thread why people want this (and I assumed git can do
this from this thread):
http://groups.google.com/group/sage-devel/browse_thread/thread/7b116d902ee20d9c/
Thanks,
Ondrej
From: Ondrej Certik <hidden> Date: 2016-06-15 22:45:41
On Tue, Nov 25, 2008 at 6:31 PM, Michael J Gruber
[off-list ref] wrote:
Ondrej Certik venit, vidit, dixit 25.11.2008 17:44:
quoted
Hi,
I would like to export our whole git repository to patches, and then
reconstruct it again from scratch. Following the man page of "git
fast-export":
$ git clone git://git.sympy.org/sympy-full-history-20081023.git
$ cd sympy-full-history-20081023
$ git fast-export --all --export-marks=marks > patches
$ cd ..
$ mkdir sympy-new
$ cd sympy-new
$ git init
$ git fast-import --export-marks=marks < ../sympy-full-history-20081023/patches
git-fast-import statistics:
---------------------------------------------------------------------
Alloc'd objects: 25000
Total objects: 21355 ( 144 duplicates )
blobs : 8009 ( 0 duplicates 4529 deltas)
trees : 10627 ( 144 duplicates 9189 deltas)
commits: 2719 ( 0 duplicates 0 deltas)
tags : 0 ( 0 duplicates 0 deltas)
Total branches: 21 ( 26 loads )
marks: 1048576 ( 10728 unique )
atoms: 726
Memory total: 2880 KiB
pools: 2098 KiB
objects: 781 KiB
---------------------------------------------------------------------
pack_report: getpagesize() = 4096
pack_report: core.packedGitWindowSize = 33554432
pack_report: core.packedGitLimit = 268435456
pack_report: pack_used_ctr = 40706
pack_report: pack_mmap_calls = 2791
pack_report: pack_open_windows = 1 / 2
pack_report: pack_mapped = 26177739 / 35513414
---------------------------------------------------------------------
However, the repository is very different to the original one. It
contains only 191 patches:
$ git log --pretty=oneline | wc -l
191
and it only contains couple files. Compare this with the original repository:
$ git log --pretty=oneline | wc -l
2719
I get the same stats (with the dups) but a perfect rev count, when I use
git log --all. The reason is that the history in the imported repo is
disconnected at various places (at tagging commits)! Your command counts
only the revs backwards to the first "disconnection".
You are right! I didn't know about "git log --all".
So, the real issue is: Why has the result these cuts in the history?
Yes, I would like to know this too. E.g. if it is a problem with our
repository, or a problem in git, or whether it is just not supposed to
work.
I don't know, I just noticed that turning on rename and copy detection
makes git-fast-import crash, which shouldn't happen either. Something's
not right here. CC'ing the authors of im- and export.
BTW: Maybe you can accomplish what you want with different means? Why
export|import directly to git?
I just answered this in my other email. Basically there are
workarounds, but I would feel safe if I can (correctly) reconstruct
the whole git repository from a human readable patches.
Ondrej
On Tue, Nov 25, 2008 at 9:41 PM, Miklos Vajna [off-list ref] wrote:
quoted
On Tue, Nov 25, 2008 at 06:31:41PM +0100, Michael J Gruber [off-list ref] wrote:
quoted
I don't know, I just noticed that turning on rename and copy detection
makes git-fast-import crash, which shouldn't happen either. Something's
not right here. CC'ing the authors of im- and export.
Could you please write a testcase that reproduces your problem?
quoted
Why export|import directly to git?
I guess he did not know about filter-branch. :)
I know about filter-branch (but I am not sure it can do what I want).
I made a mistake of not explaining what I want, instead I suggested (a
possibly wrong) solution. I want to export the whole git repository as
a set of human readable patches, that can be assembled back into a git
repository (with the same hashes as the original one)
the same hashes on the file is easy, the same hashes on the commits is
extremely hard. which is it that you are looking for.
that being said, the test of being able to do a export|import is a good
one to test that the export format and import format actually match.
David Lang
if needed. The
reason I want that is that if we later decide to switch to another
VCS, we have all the information to reproduce the repository. Another
reason is to be sure that we know all the sources that are needed to
construct the repository, e.g. that there are no binary blobs
(possibly containing malicious code). Another reason I want that is to
be able to rewrite the history, in particular, we have one Mercurial
repository with some old history and another Mercurial history with a
newer history and I just want to concatenate them together into one
git repository.
In each case I know several workarounds, but if there is a way to just
convert the whole git repository into a set of patches and (and be
able to convert everything back including the same hashes), then it'd
be awesome.
See also this thread why people want this (and I assumed git can do
this from this thread):
http://groups.google.com/group/sage-devel/browse_thread/thread/7b116d902ee20d9c/
Thanks,
Ondrej
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
From: Johannes Schindelin <hidden> Date: 2016-06-15 22:45:41
Hi,
On Tue, 25 Nov 2008, Ondrej Certik wrote:
I would like to export our whole git repository to patches, and then
reconstruct it again from scratch. Following the man page of "git
fast-export":
[...]
However, the repository is very different to the original one. It
contains only 191 patches:
Can you try again with a Git version that contains the commit
2075ffb5(fast-export: use an unsorted string list for extra_refs)?
Ciao,
Dscho
From: Ondrej Certik <hidden> Date: 2016-06-15 22:45:41
On Wed, Nov 26, 2008 at 1:14 AM, Johannes Schindelin
[off-list ref] wrote:
Hi,
On Tue, 25 Nov 2008, Ondrej Certik wrote:
quoted
I would like to export our whole git repository to patches, and then
reconstruct it again from scratch. Following the man page of "git
fast-export":
[...]
However, the repository is very different to the original one. It
contains only 191 patches:
Can you try again with a Git version that contains the commit
2075ffb5(fast-export: use an unsorted string list for extra_refs)?
I tried the next branch:
$ git --version
git version 1.6.0.4.1060.g9433b
that contains the 2075ffb5 patch. I haven't observed any change ---
the "git log" still only shows 191 commits (git log --all shows
everything).
Ondrej
From: Ondrej Certik <hidden> Date: 2016-06-15 22:45:41
On Wed, Nov 26, 2008 at 10:35 AM, Ondrej Certik [off-list ref] wrote:
On Wed, Nov 26, 2008 at 1:14 AM, Johannes Schindelin
[off-list ref] wrote:
quoted
Hi,
On Tue, 25 Nov 2008, Ondrej Certik wrote:
quoted
I would like to export our whole git repository to patches, and then
reconstruct it again from scratch. Following the man page of "git
fast-export":
[...]
However, the repository is very different to the original one. It
contains only 191 patches:
Can you try again with a Git version that contains the commit
2075ffb5(fast-export: use an unsorted string list for extra_refs)?
I tried the next branch:
$ git --version
git version 1.6.0.4.1060.g9433b
that contains the 2075ffb5 patch. I haven't observed any change ---
the "git log" still only shows 191 commits (git log --all shows
everything).
I deleted all tags and then fast-exported and imported, now all the
commits show in "git log", however, the patches are wrongly connected.
Basically, both repositories are identical (including hashes) up to
this commit:
d717177d4 (fixed downloads instructions in the README and a typo)
However, the original repo (sympy-full-history-20081023) contains 3
children at this commit:
Parent: fecac34251934e98a05631440d3ce151585f2391 (David added to credits)
Child: 03ccb60798d62f94ac9d2ec9472dc7333f67b420 (Allow to specify
line width in 2D plotting.)
Child: 203124d834488781db5429d941eeb60e396990c8 (credits improvements)
Child: 77146885f1b7aa49184f27c2297488c3d1201106 (Speed "import sympy"
up as in the last release.)
but the newly created repository only 2:
Parent: fecac34251934e98a05631440d3ce151585f2391 (David added to credits)
Child: 203124d834488781db5429d941eeb60e396990c8 (credits improvements)
Child: 77146885f1b7aa49184f27c2297488c3d1201106 (Speed "import sympy"
up as in the last release.)
And from that point on, the hashes mishmatch and sometimes the commits
are just wrongly connected (e.g. for example d2dc6b3's parent is
0adafe3, but 0adafe3 was committed half a year later after
d2dc6b3...), so it's a mess. Also the checkouted files are not
complete.
Now, if you look at the "patches" file, to which I saved the results
of "git fast-export", you can find that the commit d717177d4 (fixed
downloads instructions in the README and a typo) has the mark :6540,
and if you search for this mark in the patches file, you can only find
2 children:
commit refs/heads/master
mark :6542
author Ondrej Certik [off-list ref] 1198803347 +0100
committer Ondrej Certik [off-list ref] 1198803347 +0100
data 21
credits improvements
from :6540
M 100644 :6541 README
and:
commit refs/heads/master
mark :6551
author Ondrej Certik [off-list ref] 1198951670 +0100
committer Ondrej Certik [off-list ref] 1198951670 +0100
data 48
Speed "import sympy" up as in the last release.
from :6540
M 100644 :6550 sympy/printing/preview.py
however, the third child doesn't contain the "from :6540":
commit refs/heads/master
mark :24
author Ondrej Certik [off-list ref] 1198798384 +0100
committer Ondrej Certik [off-list ref] 1198798384 +0100
data 101
Allow to specify line width in 2D plotting.
So imho that's a bug in git fast-export. What do you think?
Ondrej
From: Michael J Gruber <hidden> Date: 2016-06-15 22:45:41
Johannes Schindelin venit, vidit, dixit 26.11.2008 01:14:
Hi,
On Tue, 25 Nov 2008, Ondrej Certik wrote:
quoted
I would like to export our whole git repository to patches, and then
reconstruct it again from scratch. Following the man page of "git
fast-export":
[...]
However, the repository is very different to the original one. It
contains only 191 patches:
Can you try again with a Git version that contains the commit
2075ffb5(fast-export: use an unsorted string list for extra_refs)?
With that commit cherry-picked onto today's master I get the same effect:
A) git fast-import crashes with dump generated by git fast-export -M -C
B) git fast-import produces disconnected DAG with dump generated by git
fast-export
With git 1.5.4, A) is good but B) is still bad.
Bisecting gives me (after remembering to put make in the run script,
uhm...)...
...
...tadah:
ae7c5dcef92d46cfc8987fde2c264614fe475bd1 is first bad commit
commit ae7c5dcef92d46cfc8987fde2c264614fe475bd1
Author: Alexander Gavrilov [off-list ref]
Date: Sun Jul 27 00:52:54 2008 +0400
Support copy and rename detection in fast-export.
Oh well :(
Michael
From: Johannes Schindelin <hidden> Date: 2016-06-15 22:45:41
Hi Ondrej & Michael,
On Wed, 26 Nov 2008, Michael J Gruber wrote:
Johannes Schindelin venit, vidit, dixit 26.11.2008 01:14:
quoted
Can you try again with a Git version that contains the commit
2075ffb5(fast-export: use an unsorted string list for extra_refs)?
Okay, so both of your use cases seem to be real bugs in fast-export. May
I respectfully submit a request for a test script (as patch to
t/t9301-fast-export.sh) which is as short as possible and shows the
respective bugs?
My Git time budget these days is almost negative, so I will not be able to
work on these issues without having a small and concise example.
Thanks so much,
Dscho
From: Ondrej Certik <hidden> Date: 2016-06-15 22:45:41
On Wed, Nov 26, 2008 at 1:46 PM, Johannes Schindelin
[off-list ref] wrote:
Hi Ondrej & Michael,
On Wed, 26 Nov 2008, Michael J Gruber wrote:
quoted
Johannes Schindelin venit, vidit, dixit 26.11.2008 01:14:
quoted
Can you try again with a Git version that contains the commit
2075ffb5(fast-export: use an unsorted string list for extra_refs)?
Okay, so both of your use cases seem to be real bugs in fast-export. May
I respectfully submit a request for a test script (as patch to
t/t9301-fast-export.sh) which is as short as possible and shows the
respective bugs?
My Git time budget these days is almost negative, so I will not be able to
work on these issues without having a small and concise example.
I'll do my best and try to create a simple example. I'll report back
if I have something.
Ondrej
From: Michael J Gruber <hidden> Date: 2016-06-15 22:45:41
Ondrej Certik venit, vidit, dixit 26.11.2008 11:18:
On Wed, Nov 26, 2008 at 10:35 AM, Ondrej Certik [off-list ref] wrote:
quoted
On Wed, Nov 26, 2008 at 1:14 AM, Johannes Schindelin
[off-list ref] wrote:
quoted
Hi,
On Tue, 25 Nov 2008, Ondrej Certik wrote:
quoted
I would like to export our whole git repository to patches, and then
reconstruct it again from scratch. Following the man page of "git
fast-export":
[...]
However, the repository is very different to the original one. It
contains only 191 patches:
Can you try again with a Git version that contains the commit
2075ffb5(fast-export: use an unsorted string list for extra_refs)?
I tried the next branch:
$ git --version
git version 1.6.0.4.1060.g9433b
that contains the 2075ffb5 patch. I haven't observed any change ---
the "git log" still only shows 191 commits (git log --all shows
everything).
I deleted all tags and then fast-exported and imported, now all the
commits show in "git log", however, the patches are wrongly connected.
Basically, both repositories are identical (including hashes) up to
this commit:
d717177d4 (fixed downloads instructions in the README and a typo)
However, the original repo (sympy-full-history-20081023) contains 3
children at this commit:
There's some nice 3 way branching and double 2 way merging going on. I
cut out the interesting part of the graph, making d717177d4 and
6e869485f parentless. The resulting mini DAG is reproduced correctly by
export|import, even with -M -C.
Michael
From: Ondrej Certik <hidden> Date: 2016-06-15 22:45:41
On Wed, Nov 26, 2008 at 4:35 PM, Michael J Gruber
[off-list ref] wrote:
Ondrej Certik venit, vidit, dixit 26.11.2008 11:18:
quoted
On Wed, Nov 26, 2008 at 10:35 AM, Ondrej Certik [off-list ref] wrote:
quoted
On Wed, Nov 26, 2008 at 1:14 AM, Johannes Schindelin
[off-list ref] wrote:
quoted
Hi,
On Tue, 25 Nov 2008, Ondrej Certik wrote:
quoted
I would like to export our whole git repository to patches, and then
reconstruct it again from scratch. Following the man page of "git
fast-export":
[...]
However, the repository is very different to the original one. It
contains only 191 patches:
Can you try again with a Git version that contains the commit
2075ffb5(fast-export: use an unsorted string list for extra_refs)?
I tried the next branch:
$ git --version
git version 1.6.0.4.1060.g9433b
that contains the 2075ffb5 patch. I haven't observed any change ---
the "git log" still only shows 191 commits (git log --all shows
everything).
I deleted all tags and then fast-exported and imported, now all the
commits show in "git log", however, the patches are wrongly connected.
Basically, both repositories are identical (including hashes) up to
this commit:
d717177d4 (fixed downloads instructions in the README and a typo)
However, the original repo (sympy-full-history-20081023) contains 3
children at this commit:
There's some nice 3 way branching and double 2 way merging going on. I
cut out the interesting part of the graph, making d717177d4 and
6e869485f parentless. The resulting mini DAG is reproduced correctly by
export|import, even with -M -C.
I am also trying to make the example simpler. I tried to squash the
first uninteresting ~1500 commits into one, but "git rebase -i"
uterrly fails after squashing about 600 commits. Still investigating.
Ondrej
From: Johannes Schindelin <hidden> Date: 2016-06-15 22:45:41
Hi,
On Wed, 26 Nov 2008, Ondrej Certik wrote:
I am also trying to make the example simpler. I tried to squash the
first uninteresting ~1500 commits into one, but "git rebase -i" uterrly
fails after squashing about 600 commits. Still investigating.
1500... wow.
The best idea would probably be to just "edit" the first, delete the rest
of the 1500, and then 'git read-tree -u -m <last-of-the-1500-commits>"' on
the command line (when git rebase stops after the "edit" command).
rebase -i was _never_ meant for such _massive_ interactions; that's just
too much for a shell script.
Ciao,
Dscho
From: Johannes Sixt <hidden> Date: 2016-06-15 22:45:41
Ondrej Certik schrieb:
I am also trying to make the example simpler. I tried to squash the
first uninteresting ~1500 commits into one, but "git rebase -i"
uterrly fails after squashing about 600 commits. Still investigating.
Don't use rebase. Set a graft and rewrite the history:
$ echo $(git rev-parse HEAD) $(git rev-parse HEAD~1500) >> \
.git/info/grafts
Assuming "first 1500" means the "most recent 1500" commits. But you get
the idea. You can truncate history as well by omitting the second SHA1.
It's very convenient to keep gitk open and File->Reload after each graft
that you set.
When you're done with setting grafts:
$ git filter-branch -f --tag-name-filter cat -- --all
(You are doing this on a copy of your repository, don't you?)
-- Hannes
From: Michael J Gruber <hidden> Date: 2016-06-15 22:45:41
Johannes Schindelin venit, vidit, dixit 26.11.2008 17:40:
Hi,
On Wed, 26 Nov 2008, Ondrej Certik wrote:
quoted
I am also trying to make the example simpler. I tried to squash the
first uninteresting ~1500 commits into one, but "git rebase -i" uterrly
fails after squashing about 600 commits. Still investigating.
1500... wow.
The best idea would probably be to just "edit" the first, delete the rest
of the 1500, and then 'git read-tree -u -m <last-of-the-1500-commits>"' on
the command line (when git rebase stops after the "edit" command).
rebase -i was _never_ meant for such _massive_ interactions; that's just
too much for a shell script.
Or chop the DAG with grafts.
Removing the tags one by one I noticed that for several of them, removal
of the tag increases the number of commits on the connected DAG
component containing master (in the ex/imported repo), and that one
reaches the correct number with still a few tags left in there. Yet, the
topology is wrong in several places; I think all of them can be
attributed to missing parent info (which even creates new roots in some
places).
Looking at the source I suspect that fast-export fails to denote
parenthood in the case of yet unmarked parents (last for-loop of
handle_commit() in builtin_fast_export.c). But I don't really know that
code at all.
Michael
P.S.: That git repo itself is a product of
hg-fast-export|git-fast-import, right?
From: Johannes Schindelin <hidden> Date: 2016-06-15 22:45:41
Hi,
On Wed, 26 Nov 2008, Michael J Gruber wrote:
Looking at the source I suspect that fast-export fails to denote
parenthood in the case of yet unmarked parents (last for-loop of
handle_commit() in builtin_fast_export.c). But I don't really know that
code at all.
I strongly doubt so. Noticed the use of has_unshown_parent(commit) in
both cases before calling handle_commit()?
In any case, here is a script that I wrote _long_ time ago, to be able to
reconstruct history from the output of "git rev-list --all --parents".
Maybe this helps you in reconstructing something that is handled
incorrectly by fast-export | fast-import, but is lighter than a full-blown
repository.
-- snip --
#!/bin/sh
# Given the output of git-rev-list, this reconstructs the DAG of the
history
i=0
tac | while read rev parents; do
let i=$i+1
echo $i > a1
git add a1
tree=$(git write-tree)
parents="$(for parent in $parents
do
echo -n "-p $(git rev-parse sp-$parent) "
done)"
commit=$(echo "$rev $i" | git commit-tree $tree $parents)
git tag sp-$rev $commit
done
-- snap --
Ciao,
Dscho
--
|
Ceci n'est pas une pipe
From: Ondrej Certik <hidden> Date: 2016-06-15 22:45:41
On Wed, Nov 26, 2008 at 5:40 PM, Johannes Schindelin
[off-list ref] wrote:
Hi,
On Wed, 26 Nov 2008, Ondrej Certik wrote:
quoted
I am also trying to make the example simpler. I tried to squash the
first uninteresting ~1500 commits into one, but "git rebase -i" uterrly
fails after squashing about 600 commits. Still investigating.
1500... wow.
The best idea would probably be to just "edit" the first, delete the rest
of the 1500, and then 'git read-tree -u -m <last-of-the-1500-commits>"' on
the command line (when git rebase stops after the "edit" command).
That worked, thanks! My original repo:
A -- B -- ... --- D --- E --- ...
where E and the rest of the commits (there are branches and merges in
there) are the ones that I need to preserve, but all the commits
between B and D can be squashed (~1500 of them). So I created a
branch:
A -- B -- ... --- D
then squashed the commits using the technique you described above, so
now I have:
A -- BD --
and now I would like to append "E -- ..." to it -- is there any way to
do that? I tried rebase, but that destroys all the branches and merges
and those are necessary to reproduce the fast-export bug.
rebase -i was _never_ meant for such _massive_ interactions; that's just
too much for a shell script.
In fact, I think it would work, but there is probably another bug,
that I am hitting, maybe due to whitespace problems --- in the
original repository, the patches are linear, but when I create a
branch before the failing patch and then cherry-pick it (that should
work), it fails and creates conflicts. The same behavior is with git
rebase. So I'll investigate more and report it in a separate thread,
as it is not related to fast-export.
Ondrej
From: Ondrej Certik <hidden> Date: 2016-06-15 22:45:42
Hi Johannes!
On Wed, Nov 26, 2008 at 5:35 PM, Johannes Sixt [off-list ref] wrote:
Ondrej Certik schrieb:
quoted
I am also trying to make the example simpler. I tried to squash the
first uninteresting ~1500 commits into one, but "git rebase -i"
uterrly fails after squashing about 600 commits. Still investigating.
Don't use rebase. Set a graft and rewrite the history:
$ echo $(git rev-parse HEAD) $(git rev-parse HEAD~1500) >> \
.git/info/grafts
Assuming "first 1500" means the "most recent 1500" commits. But you get
the idea. You can truncate history as well by omitting the second SHA1.
It's very convenient to keep gitk open and File->Reload after each graft
that you set.
When you're done with setting grafts:
$ git filter-branch -f --tag-name-filter cat -- --all
Indeed, this seems to be working robustly. Thanks!
(You are doing this on a copy of your repository, don't you?)
Yes.
I spent the whole today trying to isolate the bug, but so far I
haven't succeeded. Unfortunately, I need to work on other things now,
so I am postponing this to some later time. The repository that
reproduces it will stay online, so anyone feel free to produce a nice
and simple (failing) test for the bug.
Thanks,
Ondrej
From: Andreas Ericsson <hidden> Date: 2016-06-15 22:45:42
Ondrej Certik wrote:
On Wed, Nov 26, 2008 at 5:40 PM, Johannes Schindelin
[off-list ref] wrote:
quoted
Hi,
On Wed, 26 Nov 2008, Ondrej Certik wrote:
quoted
I am also trying to make the example simpler. I tried to squash the
first uninteresting ~1500 commits into one, but "git rebase -i" uterrly
fails after squashing about 600 commits. Still investigating.
1500... wow.
The best idea would probably be to just "edit" the first, delete the rest
of the 1500, and then 'git read-tree -u -m <last-of-the-1500-commits>"' on
the command line (when git rebase stops after the "edit" command).
That worked, thanks! My original repo:
A -- B -- ... --- D --- E --- ...
where E and the rest of the commits (there are branches and merges in
there) are the ones that I need to preserve, but all the commits
between B and D can be squashed (~1500 of them). So I created a
branch:
A -- B -- ... --- D
then squashed the commits using the technique you described above, so
now I have:
A -- BD --
and now I would like to append "E -- ..." to it -- is there any way to
do that? I tried rebase, but that destroys all the branches and merges
and those are necessary to reproduce the fast-export bug.
git rebase -p
If your git is old, you'll need
git rebase -i -p
--
Andreas Ericsson andreas.ericsson@op5.se
OP5 AB www.op5.se
Tel: +46 8-230225 Fax: +46 8-230231
From: Alexander Gavrilov <hidden> Date: 2016-06-15 22:45:44
On Wednesday 26 November 2008 20:08:54 Johannes Schindelin wrote:
On Wed, 26 Nov 2008, Michael J Gruber wrote:
quoted
Looking at the source I suspect that fast-export fails to denote
parenthood in the case of yet unmarked parents (last for-loop of
handle_commit() in builtin_fast_export.c). But I don't really know that
code at all.
I strongly doubt so. Noticed the use of has_unshown_parent(commit) in
both cases before calling handle_commit()?
In any case, here is a script that I wrote _long_ time ago, to be able to
reconstruct history from the output of "git rev-list --all --parents".
Maybe this helps you in reconstructing something that is handled
incorrectly by fast-export | fast-import, but is lighter than a full-blown
repository.
Today I had time to investigate this problem, and found:
1) The root of the problem is that fast-export really wants to walk
revisions in topological order, but actually receives them in date
order. While it is usually a good guess at topology, this repository
contains some children that are older than their parent commits,
e.g. see dd22c7d51a4debf18a3b2e35c61a1fec0175e4e0
2) It tries to fix minor deviations by checking the SHOWN flag.
However, it still breaks in two ways:
a) SHOWN is apparently set by simply walking the commits, so
if the parent was earlier encountered on a different branch of
the DAG, it will be handled as already shown. Basically, this
check is only good to determine if we reached the end of the
chain, and should start to backtrack.
b) If I modify the code to use a completely separate flag, it still
doesn't work, because the commits are placed on the stack in
the wrong order, so handle_tail gets stuck, and fails to unwind
it completely.
So, apparently, the only way to fix it is to require topological order:
@@ -490,6 +490,7 @@ int cmd_fast_export(int argc, const char **argv, const char *prefix)git_config(git_default_config,NULL);init_revisions(&revs,prefix);+revs.topo_order=1;/* force topological ordering */argc=setup_revisions(argc,argv,&revs,NULL);argc=parse_options(argc,argv,options,fast_export_usage,0);if(argc>1)
(It is also possible to specify --topo-order on the command line)
As for the failure to import the output of fast-export with copy detection,
it is a natural consequence of a messed up order of commits, because
copy and move commands depend on the original file being there.
The attachment contains a (proper) export of a simplified sample of the
structure around dd22c7d51a4d, which clearly reproduces the problem
because all of the timestamps were made identical as a side effect of
simplification. By crafting timestamps it is probably possible to minimize
it even further.
Alexander
From: Johannes Schindelin <hidden> Date: 2016-06-15 22:45:44
Hi,
On Sun, 7 Dec 2008, Alexander Gavrilov wrote:
On Wednesday 26 November 2008 20:08:54 Johannes Schindelin wrote:
quoted
On Wed, 26 Nov 2008, Michael J Gruber wrote:
quoted
Looking at the source I suspect that fast-export fails to denote
parenthood in the case of yet unmarked parents (last for-loop of
handle_commit() in builtin_fast_export.c). But I don't really know
that code at all.
I strongly doubt so. Noticed the use of has_unshown_parent(commit) in
both cases before calling handle_commit()?
In any case, here is a script that I wrote _long_ time ago, to be able
to reconstruct history from the output of "git rev-list --all
--parents". Maybe this helps you in reconstructing something that is
handled incorrectly by fast-export | fast-import, but is lighter than
a full-blown repository.
Today I had time to investigate this problem, and found:
1) The root of the problem is that fast-export really wants to walk
revisions in topological order, but actually receives them in date
order.
Indeed. Can you submit this patch with a proper commit message, adding a
test for the issue by setting a bogus GIT_COMMITTER_DATE explicitly?
Thanks,
Dscho
From: Ondrej Certik <hidden> Date: 2016-06-15 22:46:42
On Wed, Nov 26, 2008 at 9:35 AM, Johannes Sixt [off-list ref] wrote:
Ondrej Certik schrieb:
quoted
I am also trying to make the example simpler. I tried to squash the
first uninteresting ~1500 commits into one, but "git rebase -i"
uterrly fails after squashing about 600 commits. Still investigating.
Don't use rebase. Set a graft and rewrite the history:
$ echo $(git rev-parse HEAD) $(git rev-parse HEAD~1500) >> \
.git/info/grafts
Assuming "first 1500" means the "most recent 1500" commits. But you get
the idea. You can truncate history as well by omitting the second SHA1.
It's very convenient to keep gitk open and File->Reload after each graft
that you set.
When you're done with setting grafts:
$ git filter-branch -f --tag-name-filter cat -- --all
(You are doing this on a copy of your repository, don't you?)
Thanks for the tip with grafts. I tried that on some other repository
when I needed to truncate the history and it works like a charm.
Ondrej