From: Colin Walters <hidden> Date: 2016-06-15 23:03:56
Hi,
TL;DR: Let's define a standard for embedding stronger checksums in tags and commit messages:
https://github.com/cgwalters/homegit/blob/master/bin/git-evtag
I think tarballs should go away as a source distribution mechanism in favor of pure git. I won't go into too many details of the "why" here (hopefully most of you agree!) but that's the background.
Now, there are a few things that the classical tarball model provides:
- Version numbers compatible with dpkg/rpm/etc
-> Do the same with your tag names, and use a well known scheme like "v$VERSION"
- The assumption that this source has been run through some tests
-> Broken assumption, and regardless you want to rerun tests downstream
- Hosting providers typically offer a strong checksum over the entire source
-> The topic of this post
The above strawman code allows embedding the SHA256(git archive | tar). Now,
in order to make this work, the byte output of "git archive" must never change in the
future. I'm not sure how valid an assumption this is. Timestamps are set to the
commit timestamp, but I could imagine someone wanting to come along later
and tweak the output to be compatible with some variant of tar or something.
We could define the checksum to be over the stream of raw objects, sorted by their checksum,
and that way be independent of archiving format variations.
Is there agreement that something like this makes sense in the git core? Does the
concept make sense? Does anything like this exist today? Other thoughts/objections?
From: brian m. carlson <hidden> Date: 2016-06-15 23:03:56
On Sat, Feb 28, 2015 at 09:48:05AM -0500, Colin Walters wrote:
The above strawman code allows embedding the SHA256(git archive | tar). Now,
in order to make this work, the byte output of "git archive" must never change in the
future. I'm not sure how valid an assumption this is. Timestamps are set to the
commit timestamp, but I could imagine someone wanting to come along later
and tweak the output to be compatible with some variant of tar or something.
This is not a safe assumption. Unfortunately, kernel.org assumed that
it was the case, and a change broke it. Let's please not make more code
that does that.
We could define the checksum to be over the stream of raw objects, sorted by their checksum,
and that way be independent of archiving format variations.
This would be a much better idea, assuming you mean "raw git objects".
For cryptographic purposes, it's important to make the item boundaries
unambiguous, which is usually done using the length. Since the raw git
objects include the length, this is sufficient.
If you don't make the boundaries unambiguous, you get the problem you
have with v3 OpenPGP keys, where somebody could move bytes from one
value to another, creating a different key, but with the same
fingerprint (hash value).
--
brian m. carlson / brian with sandals: Houston, Texas, US
+1 832 623 2791 | http://www.crustytoothpaste.net/~bmc | My opinion only
OpenPGP: RSA v4 4096b: 88AC E9B2 9196 305B A994 7552 F1BA 225C 0223 B187
Is there a point to including a different checksum inside
a git tag? If someone can break the SHA-1 checksum
in the repository then the recorded SHA-256 checksum can
be changed. In other words, wouldn't you be just as well
off handing someone a SHA-1 commit id?
If you can guard the SHA-256 with a signature, you can
do the same thing to the SHA-1. Or the tarball for that matter.
Unrelatedly, your assumptions:
Tar balls have too many degrees of freedom to rely on them
being created identically in the future.
- The assumption that this source has been run through some tests
A perfectly valid assumption for some build systems, notably
autotools. "make distcheck" is the only way my tarballs get
made and they only get made when the checks succeed.
(If your point was that many projects have too few tests,
well, then I agree.)
M.
On Sat, Feb 28, 2015 at 9:48 AM, Colin Walters [off-list ref] wrote:
Hi,
TL;DR: Let's define a standard for embedding stronger checksums in tags and commit messages:
https://github.com/cgwalters/homegit/blob/master/bin/git-evtag
I think tarballs should go away as a source distribution mechanism in favor of pure git. I won't go into too many details of the "why" here (hopefully most of you agree!) but that's the background.
Now, there are a few things that the classical tarball model provides:
- Version numbers compatible with dpkg/rpm/etc
-> Do the same with your tag names, and use a well known scheme like "v$VERSION"
- The assumption that this source has been run through some tests
-> Broken assumption, and regardless you want to rerun tests downstream
- Hosting providers typically offer a strong checksum over the entire source
-> The topic of this post
The above strawman code allows embedding the SHA256(git archive | tar). Now,
in order to make this work, the byte output of "git archive" must never change in the
future. I'm not sure how valid an assumption this is. Timestamps are set to the
commit timestamp, but I could imagine someone wanting to come along later
and tweak the output to be compatible with some variant of tar or something.
We could define the checksum to be over the stream of raw objects, sorted by their checksum,
and that way be independent of archiving format variations.
Is there agreement that something like this makes sense in the git core? Does the
concept make sense? Does anything like this exist today? Other thoughts/objections?
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
From: Colin Walters <hidden> Date: 2016-06-15 23:03:57
On Sat, Feb 28, 2015, at 03:34 PM, Morten Welinder wrote:
Is there a point to including a different checksum inside
a git tag? If someone can break the SHA-1 checksum
in the repository then the recorded SHA-256 checksum can
be changed. In other words, wouldn't you be just as well
off handing someone a SHA-1 commit id?
The issue is more about what the checksum covers, as
well as its strength. Git uses a hash tree, which means
that an attacker only has to find a collision for *one* of
the objects, and the signature is still valid. And that collision
is valid for *every* commit that contains that object.
This topic has been covered elsewhere pretty extensively,
here's a link:
https://www.whonix.org/forum/index.php/topic,538.msg4278.html#msg4278
Now I think rough consensus is still that git is "secure" or
"secure enough" - but with this proposal I'm just trying
to overcome the remaining conservatism. (Also, while those
discussions were focusing on corrupting an existing repository,
the attack model of MITM also exists, and there
you don't have to worry about deltas, particularly if the
attacker's goal is to get a downstream to do a build
and thus execute their hostile code inside the downstream
network).
It's really not that expensive to do once per release,
basically free for small repositories, and for a large one like
the Linux kernel:
$ cd ~/src/linux
$ git describe
v3.19-7478-g796e1c5
$ time /bin/sh -c 'git archive --format=tar HEAD|sha256sum'
4a5c5826cea188abd52fa50c663d17ebe1dfe531109fed4ddbd765a856f1966e -
real 0m3.772s
user 0m6.132s
sys 0m0.279s
$
With this proposal, the checksum
covers an entire stream of objects for a given commit at once;
making it significantly harder to find a collision. At least as good as
checksummed tarballs, and arguably better since it's
pre-compression.
So to implement this, perhaps something like:
$ git archive --format=raw
as a base primitive, and:
$ git tag --archive-raw-checksum=SHA256 -s -m "..."
?
"git fsck" could also learn to optionally use this.
I support this proposal, as someone who no longer releases tarballs
of my software, when I can possibly avoid it. I have worried about
signed tags / commits only being a SHA1 break away from useless.
As to the implementation, checksumming the collection of raw objects is
certainly superior to tar. Colin had suggested sorting the objects by
checksum, but I don't think that is necessary. Just stream the commit
object, then its tree object, followed by the content of each object
listed in the tree, recursing into subtrees as necessary. That will be a
stable stream for a given commit, or tree.
--
see shy jo
From: Sam Vilain <hidden> Date: 2016-06-15 23:03:57
On 03/02/2015 10:12 AM, Joey Hess wrote:
I support this proposal, as someone who no longer releases tarballs
of my software, when I can possibly avoid it. I have worried about
signed tags / commits only being a SHA1 break away from useless.
As to the implementation, checksumming the collection of raw objects is
certainly superior to tar. Colin had suggested sorting the objects by
checksum, but I don't think that is necessary. Just stream the commit
object, then its tree object, followed by the content of each object
listed in the tree, recursing into subtrees as necessary. That will be a
stable stream for a given commit, or tree.
I would really just do it exactly the same way that git does: checksum
the objects including their headers with the new hashes. I have a hazy
recollection of what it would take to replace SHA-1 in git with
something else; it should be possible (though tricky) to do it lazily,
where a tree entry has bits (eg, some of the currently unused file mode
bits) to denotes which hash algorithm is in use for the entry. However
I don't think that got past idea stage...
Sam
On Tue, Mar 3, 2015 at 1:12 AM, Joey Hess [off-list ref] wrote:
I support this proposal, as someone who no longer releases tarballs
of my software, when I can possibly avoid it. I have worried about
signed tags / commits only being a SHA1 break away from useless.
As to the implementation, checksumming the collection of raw objects is
certainly superior to tar. Colin had suggested sorting the objects by
checksum, but I don't think that is necessary. Just stream the commit
object, then its tree object, followed by the content of each object
listed in the tree, recursing into subtrees as necessary. That will be a
stable stream for a given commit, or tree.
It could be simplified a bit by using ls-tree -r (so you basically
have a single big tree). Then hash commit, ls-tree -r output and all
blobs pointed by ls-tree in listed order.
--
Duy
[time passes]
I finally had a bit of time to pick this back up again in:
https://github.com/cgwalters/git-evtag
It should address the core concern here about stability of `git archive`.
I prototyped it out with libgit2 because it was easier, and I'd like actually to be able to use this with older versions of git.
But I think the next steps here are:
- Validate the core design
* Tree walking order
* Submodule recursion
* Use of SHA512
- Standardize it
(Would like to see at least a stupid slow shell script implementation to cross-validate)
- Add it as an option to `git tag`?
Longer term:
- Support adding `Git-EVTag` as a git note, so I can retroactively add stronger
checksums to older git repositories
- Anything else?