Hi.
I can't get the same commit hashes after fast-export and then fast-import of
this repository without any edits https://github.com/simons-public/protonfixes
I have no idea what causes this, and how to prevent it from happening. Are
there any workarounds?
What did you do before the bug happened? (Steps to reproduce your issue)
#!/bin/bash
git clone https://github.com/simons-public/protonfixes.git
git -C protonfixes log --format=oneline | tail -n 4
git init protoimported
git -C protonfixes fast-export --all --reencode=no | (cd
protoimported && git fast-import)
git -C protoimported log --format=oneline | tail -n 4
What did you expect to happen? (Expected behavior)
Expect imported repo to match exported.
What happened instead? (Actual behavior)
All hashes are different, the exported repo diverged on the second commit.
What's different between what you expected and what actually happened?
The log of hashes from initial repo:
+ git -C protonfixes log --format=oneline
+ tail -n 4
1c0cf2c8e742e673dba9fd1a09afd12a25c25571 Update README.md
367d61f9b2a799accbdaeed5d64f9be914ca0f7a Updated zip link
d3d24b63446c7d06586eaa51764ff0c619113f09 Update README.md
7a43ca89ff7a70127ac9ca0f10b6eaaa34f2f69c Initial commit
The log from imported repo:
+ git -C protoimported log --format=oneline
+ tail -n 4
a27ec5d2e4c562f40e693e0b4149959d2b69bf21 Update README.md
e59cf92be79c47984e9f94bfad912e5a29dfa5e0 Updated zip link
fb6498f62af783d2e943770f90bc642cf5c9ec9c Update README.md
7a43ca89ff7a70127ac9ca0f10b6eaaa34f2f69c Initial commit
[System Info]
git version:
git version 2.31.0.rc0
cpu: x86_64
built from commit: 225365fb5195e804274ab569ac3cc4919451dc7f
sizeof-long: 8
sizeof-size_t: 8
shell-path: /bin/sh
uname: Linux 5.8.0-43-generic #49-Ubuntu SMP Fri Feb 5 03:01:28 UTC 2021 x86_64
compiler info: gnuc: 10.2
libc info: glibc: 2.32
$SHELL (typically, interactive shell): /usr/bin/zsh
[Enabled Hooks]
not run from a git repository - no hooks to show
--
anatoly t.
Hi,
On Sat, Feb 27, 2021 at 4:37 AM anatoly techtonik [off-list ref] wrote:
Hi.
I can't get the same commit hashes after fast-export and then fast-import of
this repository without any edits https://github.com/simons-public/protonfixes
I have no idea what causes this, and how to prevent it from happening. Are
there any workarounds?
Your second commit is signed. Fast-export strips any extended headers
on commits, such as GPG signatures, because there's no way to keep
them in general. In the special case that you aren't making *any*
changes to the repository and will import it as-is, you could
theoretically keep the signatures, but you don't need fast-export in
such a case so no one ever bothered to implement commit signature
handling in fast-export and fast-import. If you make any changes
whatsoever to the commits before the signature (including importing
them to a different system), then the signature would be invalid.
You probably don't want to hear this, but there are no workarounds.
There are also other things that will prevent a simple fast-export |
fast-import pipeline from preserving your history as-is besides signed
commits (most of these are noted in the "Inherited Limitations"
section over at
https://htmlpreview.github.io/?https://github.com/newren/git-filter-repo/blob/docs/html/git-filter-repo.html):
* any other form of extended header; fast-export only looks for the
headers it knows and exports those
* grafts and replace objects will just get rewritten (and if they
cause any cycles, those cycles and anything depending on them are
dropped)
* commits without an author will be given one matching the committer
(hopefully you don't have these, but if you do...)
* tags that are missing a tagger are also a problem (hopefully you
don't have these, but if you do...)
* annotated or signed tags outside the refs/tags/ namespace will get
renamed weirdly
* commits by default are re-encoded into UTF-8, though I notice you
did pass --reencode=no to handle this
Hope that at least explains things for you, even if it doesn't give
you a workaround or a solution.
On Sat, Feb 27, 2021 at 8:49 PM Elijah Newren [off-list ref] wrote:
Your second commit is signed. Fast-export strips any extended headers
on commits, such as GPG signatures, because there's no way to keep
them in general.
Why is it not possible to encode them with base64 and insert into the
stream?
Is there any way to check what commits will be altered as a result of
`fast-export` and why? Right now I don't see that it is reported.
Hope that at least explains things for you, even if it doesn't give
you a workaround or a solution.
Thanks. That is very helpful to know.
The reason I am asking is because I tried to merge two repos with
`reposurgeon` which operates on `fast-export` data. It is basically
merging GitHub wiki into main repo,
After successfully merging them I still can not send a PR, because
it produces a huge amount of changes, because of the stripped info.
It can be seen here:
https://github.com/simons-public/protonfixes/compare/master...techtonik:master
I tracked this behaviour in `reposurgeon` in this issue
https://gitlab.com/esr/reposurgeon/-/issues/344
--
anatoly t.
On Sat, Feb 27, 2021 at 8:49 PM Elijah Newren [off-list ref] wrote:
quoted
Your second commit is signed. Fast-export strips any extended headers
on commits, such as GPG signatures, because there's no way to keep
them in general.
Why is it not possible to encode them with base64 and insert into the
stream?
I think Elijah means that in the general case people are using fast
export/import to export/import between different systems or in
combination with a utility like git-filter-repo.
In those cases users are also changing the content of the repository, so
the hashes will change, invalidating signatures.
But there's also cases where e.g. you don't modify the history, or only
part of it, and could then preserve these headers. I think there's no
inherent reason not to do so, just that nobody's cared enough to submit
patches etc.
Is there any way to check what commits will be altered as a result of
`fast-export` and why? Right now I don't see that it is reported.
I don't think so, but not being very familiar with fast export/import I
don't see why it shouldn't have some option to not munge data like that,
or to report it, if someone cared enough to track those issues & patch
it...
quoted
Hope that at least explains things for you, even if it doesn't give
you a workaround or a solution.
Thanks. That is very helpful to know.
The reason I am asking is because I tried to merge two repos with
`reposurgeon` which operates on `fast-export` data. It is basically
merging GitHub wiki into main repo,
After successfully merging them I still can not send a PR, because
it produces a huge amount of changes, because of the stripped info.
It can be seen here:
https://github.com/simons-public/protonfixes/compare/master...techtonik:master
I tracked this behaviour in `reposurgeon` in this issue
https://gitlab.com/esr/reposurgeon/-/issues/344
On Sun, Feb 28, 2021 at 1:34 PM Ævar Arnfjörð Bjarmason
[off-list ref] wrote:
I think Elijah means that in the general case people are using fast
export/import to export/import between different systems or in
combination with a utility like git-filter-repo.
In those cases users are also changing the content of the repository, so
the hashes will change, invalidating signatures.
But there's also cases where e.g. you don't modify the history, or only
part of it, and could then preserve these headers. I think there's no
inherent reason not to do so, just that nobody's cared enough to submit
patches etc.
Is fast-export/import the only way to filter information in `git`? Maybe there
is a slow json-export/import tool that gives a complete representation of all
events in a repository? Or API that can be used to serialize and import that
stream?
If no, then I'd like to take a look at where header filtering and serialization
takes place. My C skills are at the "hello world" level, so I am not sure I can
write a patch. But I can write the logic in Python and ask somebody to port
that.
--
anatoly t.
On Sun, Feb 28, 2021 at 11:44 PM anatoly techtonik [off-list ref] wrote:
On Sun, Feb 28, 2021 at 1:34 PM Ævar Arnfjörð Bjarmason
[off-list ref] wrote:
quoted
I think Elijah means that in the general case people are using fast
export/import to export/import between different systems or in
combination with a utility like git-filter-repo.
In those cases users are also changing the content of the repository, so
the hashes will change, invalidating signatures.
But there's also cases where e.g. you don't modify the history, or only
part of it, and could then preserve these headers. I think there's no
inherent reason not to do so, just that nobody's cared enough to submit
patches etc.
Is fast-export/import the only way to filter information in `git`? Maybe there
is a slow json-export/import tool that gives a complete representation of all
events in a repository? Or API that can be used to serialize and import that
stream?
If no, then I'd like to take a look at where header filtering and serialization
takes place. My C skills are at the "hello world" level, so I am not sure I can
write a patch. But I can write the logic in Python and ask somebody to port
that.
If you are intent on keeping signatures because you know they are
still valid, then you already know you aren't modifying any
blobs/trees/commits leading up to those signatures. If that is the
case, perhaps you should just avoid exporting the signature or
anything it depends on, and just export the stuff after that point.
You can do this with fast-export's --reference-excluded-parents option
and pass it an exclusion range. For example:
git fast-export --reference-excluded-parents ^master~5 --all
and then pipe that through fast-import.
In general, I think if fast-export or fast-import are lacking features
you want, we should add them there, but I don't see how adding
signature reading to fast-import and signature exporting to
fast-export makes sense in general. Even if you assume fast-import
can process all the bits it is sent (e.g. you extend it to support
commits without an author, tags without a tagger, signed objects, any
other extended commit headers), and even if you add flags to
fast-export to die if there are any bits it doesn't recognize and to
export all pieces of blobs/trees/tags (e.g. don't add missing authors,
don't re-encode messages in UTF-8, don't use grafts or replace
objects, keep extended headers such as signatures, etc.), then it
still couldn't possibly work in all cases in general. For example, if
you had a repository with unusual objects made by ancient or broken
git versions (such as tree entries in the wrong sort order, or tree
entries that recorded modes of 040000 instead of 40000 for trees or
something with perms other than 100644 or 100755 for files), then when
fast-import goes to recreate these objects using the canonical format
they will no longer have the same hash and your commit signatures will
get invalidated. Other git commands will also refuse to create
objects with those oddities, even if git accepts ancient objects that
have them.
So, it's basically impossible to have a "complete representation of
all events in a repository" that do what you want except for the
*original* binary format. (But if you really want to see the original
binary format, maybe `git cat-file --batch` will be handy to you.)
But I think fast-export's --reference-excluded-parents might come in
handy for you and let you do what you want.
On Sun, Feb 28, 2021 at 1:34 PM Ævar Arnfjörð Bjarmason
[off-list ref] wrote:
quoted
I think Elijah means that in the general case people are using fast
export/import to export/import between different systems or in
combination with a utility like git-filter-repo.
In those cases users are also changing the content of the repository, so
the hashes will change, invalidating signatures.
But there's also cases where e.g. you don't modify the history, or only
part of it, and could then preserve these headers. I think there's no
inherent reason not to do so, just that nobody's cared enough to submit
patches etc.
Is fast-export/import the only way to filter information in `git`? Maybe there
is a slow json-export/import tool that gives a complete representation of all
events in a repository? Or API that can be used to serialize and import that
stream?
Aside from other things mentioned & any issues in fast export/import in
this thread, if you want round-trip correctness you're not going to want
JSON-anything. It's not capable of representing arbitrary binary data.
But in any case, it's not the fast-export format that's the issue, but
how the tools in git.git are munging/rewriting/omitting the repository
data in question...
On Sun, Feb 28, 2021 at 11:44 PM anatoly techtonik [off-list ref] wrote:
quoted
On Sun, Feb 28, 2021 at 1:34 PM Ævar Arnfjörð Bjarmason
[off-list ref] wrote:
quoted
I think Elijah means that in the general case people are using fast
export/import to export/import between different systems or in
combination with a utility like git-filter-repo.
In those cases users are also changing the content of the repository, so
the hashes will change, invalidating signatures.
But there's also cases where e.g. you don't modify the history, or only
part of it, and could then preserve these headers. I think there's no
inherent reason not to do so, just that nobody's cared enough to submit
patches etc.
Is fast-export/import the only way to filter information in `git`? Maybe there
is a slow json-export/import tool that gives a complete representation of all
events in a repository? Or API that can be used to serialize and import that
stream?
If no, then I'd like to take a look at where header filtering and serialization
takes place. My C skills are at the "hello world" level, so I am not sure I can
write a patch. But I can write the logic in Python and ask somebody to port
that.
If you are intent on keeping signatures because you know they are
still valid, then you already know you aren't modifying any
blobs/trees/commits leading up to those signatures. If that is the
case, perhaps you should just avoid exporting the signature or
anything it depends on, and just export the stuff after that point.
You can do this with fast-export's --reference-excluded-parents option
and pass it an exclusion range. For example:
git fast-export --reference-excluded-parents ^master~5 --all
and then pipe that through fast-import.
In general, I think if fast-export or fast-import are lacking features
you want, we should add them there, but I don't see how adding
signature reading to fast-import and signature exporting to
fast-export makes sense in general. Even if you assume fast-import
can process all the bits it is sent (e.g. you extend it to support
commits without an author, tags without a tagger, signed objects, any
other extended commit headers), and even if you add flags to
fast-export to die if there are any bits it doesn't recognize and to
export all pieces of blobs/trees/tags (e.g. don't add missing authors,
don't re-encode messages in UTF-8, don't use grafts or replace
objects, keep extended headers such as signatures, etc.), then it
still couldn't possibly work in all cases in general. For example, if
you had a repository with unusual objects made by ancient or broken
git versions (such as tree entries in the wrong sort order, or tree
entries that recorded modes of 040000 instead of 40000 for trees or
something with perms other than 100644 or 100755 for files), then when
fast-import goes to recreate these objects using the canonical format
they will no longer have the same hash and your commit signatures will
get invalidated. Other git commands will also refuse to create
objects with those oddities, even if git accepts ancient objects that
have them.
So, it's basically impossible to have a "complete representation of
all events in a repository" that do what you want except for the
*original* binary format. (But if you really want to see the original
binary format, maybe `git cat-file --batch` will be handy to you.)
But I think fast-export's --reference-excluded-parents might come in
handy for you and let you do what you want.
...to add to that line of thinking, it's also a completely valid
technique to just completele rewrite your repository, then (re-)push the
old signed tags to refs/tags/*.
By default they won't be pulled down as they won't reference commits on
branches you're fetching, and you can also stick them somewhere else
than refs/tags/*, e.g. refs/legacy-tags/*.
None of the commit history will be the same, but the content (mostly)
will, which is usually what matters when checking out an old tag.
Of course this hack has little benefit over just keeping a foo-old.git
repo around, and moving on with new history in your new foo.git.
On Mon, Mar 1, 2021 at 12:04 PM Ævar Arnfjörð Bjarmason
[off-list ref] wrote:
On Mon, Mar 01 2021, Elijah Newren wrote:
quoted
On Sun, Feb 28, 2021 at 11:44 PM anatoly techtonik [off-list ref] wrote:
quoted
On Sun, Feb 28, 2021 at 1:34 PM Ævar Arnfjörð Bjarmason
[off-list ref] wrote:
quoted
I think Elijah means that in the general case people are using fast
export/import to export/import between different systems or in
combination with a utility like git-filter-repo.
In those cases users are also changing the content of the repository, so
the hashes will change, invalidating signatures.
But there's also cases where e.g. you don't modify the history, or only
part of it, and could then preserve these headers. I think there's no
inherent reason not to do so, just that nobody's cared enough to submit
patches etc.
Is fast-export/import the only way to filter information in `git`? Maybe there
is a slow json-export/import tool that gives a complete representation of all
events in a repository? Or API that can be used to serialize and import that
stream?
If no, then I'd like to take a look at where header filtering and serialization
takes place. My C skills are at the "hello world" level, so I am not sure I can
write a patch. But I can write the logic in Python and ask somebody to port
that.
If you are intent on keeping signatures because you know they are
still valid, then you already know you aren't modifying any
blobs/trees/commits leading up to those signatures. If that is the
case, perhaps you should just avoid exporting the signature or
anything it depends on, and just export the stuff after that point.
You can do this with fast-export's --reference-excluded-parents option
and pass it an exclusion range. For example:
git fast-export --reference-excluded-parents ^master~5 --all
and then pipe that through fast-import.
In general, I think if fast-export or fast-import are lacking features
you want, we should add them there, but I don't see how adding
signature reading to fast-import and signature exporting to
fast-export makes sense in general. Even if you assume fast-import
can process all the bits it is sent (e.g. you extend it to support
commits without an author, tags without a tagger, signed objects, any
other extended commit headers), and even if you add flags to
fast-export to die if there are any bits it doesn't recognize and to
export all pieces of blobs/trees/tags (e.g. don't add missing authors,
don't re-encode messages in UTF-8, don't use grafts or replace
objects, keep extended headers such as signatures, etc.), then it
still couldn't possibly work in all cases in general. For example, if
you had a repository with unusual objects made by ancient or broken
git versions (such as tree entries in the wrong sort order, or tree
entries that recorded modes of 040000 instead of 40000 for trees or
something with perms other than 100644 or 100755 for files), then when
fast-import goes to recreate these objects using the canonical format
they will no longer have the same hash and your commit signatures will
get invalidated. Other git commands will also refuse to create
objects with those oddities, even if git accepts ancient objects that
have them.
So, it's basically impossible to have a "complete representation of
all events in a repository" that do what you want except for the
*original* binary format. (But if you really want to see the original
binary format, maybe `git cat-file --batch` will be handy to you.)
But I think fast-export's --reference-excluded-parents might come in
handy for you and let you do what you want.
...to add to that line of thinking, it's also a completely valid
technique to just completele rewrite your repository, then (re-)push the
old signed tags to refs/tags/*.
The repository in question didn't have any signed tags, just a signed commit.
By default they won't be pulled down as they won't reference commits on
branches you're fetching, and you can also stick them somewhere else
than refs/tags/*, e.g. refs/legacy-tags/*.
None of the commit history will be the same, but the content (mostly)
will, which is usually what matters when checking out an old tag.
Of course this hack has little benefit over just keeping a foo-old.git
repo around, and moving on with new history in your new foo.git.
On Mon, Mar 1, 2021 at 11:02 PM Ævar Arnfjörð Bjarmason
[off-list ref] wrote:
Aside from other things mentioned & any issues in fast export/import in
this thread, if you want round-trip correctness you're not going to want
JSON-anything. It's not capable of representing arbitrary binary data.
Yes, binary data needs to be explicitly represented in base64 or similar
encoding. Just the same way as ordinary strings will need escape
symbols.
--
anatoly t.
On Mon, Mar 1, 2021 at 9:06 PM Elijah Newren [off-list ref] wrote:
On Sun, Feb 28, 2021 at 11:44 PM anatoly techtonik [off-list ref] wrote:
For example:
git fast-export --reference-excluded-parents ^master~5 --all
and then pipe that through fast-import.
That may come in handy, but if certain parents are excluded, it will be
impossible to find them to reference and attach branches to them.
Other git commands will also refuse to create
objects with those oddities, even if git accepts ancient objects that
have them.
Are there any `lint` commands that can detect and warn about those
oddities?
(But if you really want to see the original
binary format, maybe `git cat-file --batch` will be handy to you.)
Looks good. Is there a way to import it back? And how hard it could
be to write a parser for it? Is there a specification for its fields?
--
anatoly t.