From: Junio C Hamano <hidden> Date: 2016-06-15 22:43:55
Nicolas Pitre [off-list ref] writes:
Two things I would like to see in the next version (1.5.5) as well, for
which we could provide early warnings now:
- repack.usedeltabaseoffset defaulting to true
- pack.indexversion defaulting to 2
I think the former would be sensible, the latter I fear might be a bit
too new but I do not recall the exact version dependency.
Could you draft a patch to ReleaseNotes to explain the consequences of
these changes using ordinary user's vocabulary, like:
Starting v1.5.5, repack.usedeltabaseoffset will default to true,
which will give denser packfile (i.e. more efficient storage).
The downside is that git older than version X will not be able
to use a repository packed using this setting.
From: Nicolas Pitre <hidden> Date: 2016-06-15 22:43:59
Signed-off-by: Nicolas Pitre <redacted>
---
On Mon, 3 Dec 2007, Junio C Hamano wrote:
Nicolas Pitre [off-list ref] writes:
quoted
Two things I would like to see in the next version (1.5.5) as well, for
which we could provide early warnings now:
- repack.usedeltabaseoffset defaulting to true
- pack.indexversion defaulting to 2
I think the former would be sensible, the latter I fear might be a bit
too new but I do not recall the exact version dependency.
Could you draft a patch to ReleaseNotes to explain the consequences of
these changes using ordinary user's vocabulary, like:
Starting v1.5.5, repack.usedeltabaseoffset will default to true,
which will give denser packfile (i.e. more efficient storage).
The downside is that git older than version X will not be able
to use a repository packed using this setting.
@@ -43,6 +43,17 @@ Deprecation notices * "git peek-remote" is deprecated, as "git ls-remote" was written in C and works for all transports, and will be removed in the future.+ * From v1.5.5, the repack.usedeltabaseoffset config option will default+ to true, which will give denser packfile (i.e. more efficient storage).+ The downside is that git older than version 1.4.4 will not be able+ to directly use a repository packed using this setting.++ * From v1.5.5, the pack.indexversion config option will default to 2,+ which is slightly more efficient, and makes repacking more immune to+ data corruptions. Git older than version 1.5.2 may revert to version 1+ of the pack index with a manual "git index-pack" to be able to directly+ access corresponding pack files.+ Updates since v1.5.3 --------------------
From: Jakub Narebski <hidden> Date: 2016-06-15 22:43:59
Nicolas Pitre [off-list ref] writes:
+ * From v1.5.5, the repack.usedeltabaseoffset config option will default
+ to true, which will give denser packfile (i.e. more efficient storage).
+ The downside is that git older than version 1.4.4 will not be able
+ to directly use a repository packed using this setting.
+
+ * From v1.5.5, the pack.indexversion config option will default to 2,
+ which is slightly more efficient, and makes repacking more immune to
+ data corruptions. Git older than version 1.5.2 may revert to version 1
+ of the pack index with a manual "git index-pack" to be able to directly
+ access corresponding pack files.
Which means what? Local clone with shortcut (hardlinking and remotes)?
Dumb protocols (http, ftp, rsync)?
--
Jakub Narebski
Poland
ShadeHawk on #git
From: Nicolas Pitre <hidden> Date: 2016-06-15 22:43:59
On Fri, 14 Dec 2007, Jakub Narebski wrote:
Nicolas Pitre [off-list ref] writes:
quoted
+ * From v1.5.5, the repack.usedeltabaseoffset config option will default
+ to true, which will give denser packfile (i.e. more efficient storage).
+ The downside is that git older than version 1.4.4 will not be able
+ to directly use a repository packed using this setting.
+
+ * From v1.5.5, the pack.indexversion config option will default to 2,
+ which is slightly more efficient, and makes repacking more immune to
+ data corruptions. Git older than version 1.5.2 may revert to version 1
+ of the pack index with a manual "git index-pack" to be able to directly
+ access corresponding pack files.
Which means what? Local clone with shortcut (hardlinking and remotes)?
Dumb protocols (http, ftp, rsync)?
Right, or simply shared repo over NFS or the like.
The 1.5.5 release notes will contain a note reminding people to set the
corresponding config variables if they wish to retain the legacy
behaviors.
Nicolas
From: Joel Becker <hidden> Date: 2016-06-15 22:43:59
On Fri, Dec 14, 2007 at 08:38:51AM -0500, Nicolas Pitre wrote:
On Fri, 14 Dec 2007, Jakub Narebski wrote:
quoted
Which means what? Local clone with shortcut (hardlinking and remotes)?
Dumb protocols (http, ftp, rsync)?
Right, or simply shared repo over NFS or the like.
The 1.5.5 release notes will contain a note reminding people to set the
corresponding config variables if they wish to retain the legacy
behaviors.
We've seen that release notes are a poor way to communicate
this. What will happen to a 1.4.4 user when they try to access the
repository? Corruption, cryptic error message, or clean "this repo is
not compatible" message?
Joel
--
"Depend on the rabbit's foot if you will, but remember, it didn't
help the rabbit."
- R. E. Shay
Joel Becker
Principal Software Developer
Oracle
E-mail: joel.becker@oracle.com
Phone: (650) 506-8127
From: Nicolas Pitre <hidden> Date: 2016-06-15 22:43:59
On Fri, 14 Dec 2007, Joel Becker wrote:
On Fri, Dec 14, 2007 at 08:38:51AM -0500, Nicolas Pitre wrote:
quoted
On Fri, 14 Dec 2007, Jakub Narebski wrote:
quoted
Which means what? Local clone with shortcut (hardlinking and remotes)?
Dumb protocols (http, ftp, rsync)?
Right, or simply shared repo over NFS or the like.
The 1.5.5 release notes will contain a note reminding people to set the
corresponding config variables if they wish to retain the legacy
behaviors.
We've seen that release notes are a poor way to communicate
this. What will happen to a 1.4.4 user when they try to access the
repository? Corruption, cryptic error message, or clean "this repo is
not compatible" message?
There won't be any corruption.
In the best case there will be a message along "x is not supported by
this version of Git -- please consider upgrading". In the worst case
it'll say "x is bad".
But you know what? repositories with the change affecting 1.4.4 users
are _already_ out there and no one complained recently. Anyone pushing
changes over the native Git protocol is already using deltabaseoffset as
the native protocol negociate that capability in its handshake, and
these days we keep packs as is on the receiving side when they're large
enough.
Nicolas
From: Joel Becker <hidden> Date: 2016-06-15 22:43:59
On Fri, Dec 14, 2007 at 05:34:49PM -0500, Nicolas Pitre wrote:
On Fri, 14 Dec 2007, Joel Becker wrote:
quoted
We've seen that release notes are a poor way to communicate
this. What will happen to a 1.4.4 user when they try to access the
repository? Corruption, cryptic error message, or clean "this repo is
not compatible" message?
There won't be any corruption.
In the best case there will be a message along "x is not supported by
this version of Git -- please consider upgrading". In the worst case
it'll say "x is bad".
That would be excellent, especially the former message.
But you know what? repositories with the change affecting 1.4.4 users
are _already_ out there and no one complained recently. Anyone pushing
I did, as did people I work with. It's on git-list, even. I'm
pretty sure it corrupted too.
Joel
--
"Practice random acts of kindness and senseless acts of beauty."
Oh, and don't forget where your towel is.
Joel Becker
Principal Software Developer
Oracle
E-mail: joel.becker@oracle.com
Phone: (650) 506-8127
From: Joel Becker <hidden> Date: 2016-06-15 22:43:59
On Fri, Dec 14, 2007 at 05:46:14PM -0500, Nicolas Pitre wrote:
On Fri, 14 Dec 2007, Joel Becker wrote:
quoted
On Fri, Dec 14, 2007 at 05:34:49PM -0500, Nicolas Pitre wrote:
quoted
But you know what? repositories with the change affecting 1.4.4 users
are _already_ out there and no one complained recently. Anyone pushing
I did, as did people I work with. It's on git-list, even. I'm
pretty sure it corrupted too.
Could you please give me a reference to such message, so to verify that
we're actually talking about the same thing?
The relevant message is:
Message-ID: [off-list ref]
See the paragraphs at the bottom. The thread, started by me, begins
with:
Message-ID: [off-list ref]
Joel
--
"You must remember this:
A kiss is just a kiss,
A sigh is just a sigh.
The fundamental rules apply
As time goes by."
Joel Becker
Principal Software Developer
Oracle
E-mail: joel.becker@oracle.com
Phone: (650) 506-8127
From: Nicolas Pitre <hidden> Date: 2016-06-15 22:43:59
On Fri, 14 Dec 2007, Joel Becker wrote:
On Fri, Dec 14, 2007 at 05:46:14PM -0500, Nicolas Pitre wrote:
quoted
On Fri, 14 Dec 2007, Joel Becker wrote:
quoted
On Fri, Dec 14, 2007 at 05:34:49PM -0500, Nicolas Pitre wrote:
quoted
But you know what? repositories with the change affecting 1.4.4 users
are _already_ out there and no one complained recently. Anyone pushing
I did, as did people I work with. It's on git-list, even. I'm
pretty sure it corrupted too.
Could you please give me a reference to such message, so to verify that
we're actually talking about the same thing?
The relevant message is:
Message-ID: [off-list ref]
See the paragraphs at the bottom. The thread, started by me, begins
with:
Message-ID: [off-list ref]
I don't have such emails in my mail folders anymore.
Nicolas
From: Johannes Schindelin <hidden> Date: 2016-06-15 22:43:59
Hi,
On Fri, 14 Dec 2007, Nicolas Pitre wrote:
On Fri, 14 Dec 2007, Joel Becker wrote:
quoted
On Fri, Dec 14, 2007 at 05:46:14PM -0500, Nicolas Pitre wrote:
quoted
On Fri, 14 Dec 2007, Joel Becker wrote:
quoted
On Fri, Dec 14, 2007 at 05:34:49PM -0500, Nicolas Pitre wrote:
quoted
But you know what? repositories with the change affecting 1.4.4 users
are _already_ out there and no one complained recently. Anyone pushing
I did, as did people I work with. It's on git-list, even. I'm
pretty sure it corrupted too.
Could you please give me a reference to such message, so to verify that
we're actually talking about the same thing?
The relevant message is:
Message-ID: [off-list ref]
See the paragraphs at the bottom. The thread, started by me, begins
with:
Message-ID: [off-list ref]
I don't have such emails in my mail folders anymore.
From: Nicolas Pitre <hidden> Date: 2016-06-15 22:43:59
On Fri, 14 Dec 2007, Joel Becker wrote:
On Fri, Dec 14, 2007 at 05:46:14PM -0500, Nicolas Pitre wrote:
quoted
On Fri, 14 Dec 2007, Joel Becker wrote:
quoted
On Fri, Dec 14, 2007 at 05:34:49PM -0500, Nicolas Pitre wrote:
quoted
But you know what? repositories with the change affecting 1.4.4 users
are _already_ out there and no one complained recently. Anyone pushing
I did, as did people I work with. It's on git-list, even. I'm
pretty sure it corrupted too.
Could you please give me a reference to such message, so to verify that
we're actually talking about the same thing?
The relevant message is:
Message-ID: [off-list ref]
See the paragraphs at the bottom. The thread, started by me, begins
with:
Message-ID: [off-list ref]
OK. From those emails Junio forwarded to me, I don't see any case for
actual _corruptions_. Git does indeed refuse to work with unknown pack
index or unknown objects in a pack. Really old versions were not overly
clueful as to why they refused to work, but they should never corrupt a
pack which, for all purposes, is always read-only anyway.
Nicolas
From: Joel Becker <hidden> Date: 2016-06-15 22:44:00
On Fri, Dec 14, 2007 at 09:23:38PM -0500, Nicolas Pitre wrote:
On Fri, 14 Dec 2007, Joel Becker wrote:
quoted
The relevant message is:
Message-ID: [off-list ref]
See the paragraphs at the bottom. The thread, started by me, begins
with:
Message-ID: [off-list ref]
OK. From those emails Junio forwarded to me, I don't see any case for
actual _corruptions_. Git does indeed refuse to work with unknown pack
index or unknown objects in a pack. Really old versions were not overly
clueful as to why they refused to work, but they should never corrupt a
pack which, for all purposes, is always read-only anyway.
You may not see a case for actual corruptions, but my coworker
updated his tree on a box with 1.5.x, then tried to work on a box with
1.4.x (I think 1.4.2 back then), and ended up with a tree that was
unusable. He had to re-clone, and I think he got lucky recovering
pending changes (probably using 1.5.x on the branches with the changes,
as master was what got broken).
My point is not that change is always bad, but that we should
really look forward to what we're doing, and that "it's in the release
notes" is not sufficient if an older git gives an incomprehensible error
or a silent problem. I was responding to the cavalier statement "well,
it's in the release notes, so don't worry about older versions".
Joel
--
"Vote early and vote often."
- Al Capone
Joel Becker
Principal Software Developer
Oracle
E-mail: joel.becker@oracle.com
Phone: (650) 506-8127
From: Nicolas Pitre <hidden> Date: 2016-06-15 22:44:00
On Mon, 17 Dec 2007, Joel Becker wrote:
On Fri, Dec 14, 2007 at 09:23:38PM -0500, Nicolas Pitre wrote:
quoted
On Fri, 14 Dec 2007, Joel Becker wrote:
quoted
The relevant message is:
Message-ID: [off-list ref]
See the paragraphs at the bottom. The thread, started by me, begins
with:
Message-ID: [off-list ref]
OK. From those emails Junio forwarded to me, I don't see any case for
actual _corruptions_. Git does indeed refuse to work with unknown pack
index or unknown objects in a pack. Really old versions were not overly
clueful as to why they refused to work, but they should never corrupt a
pack which, for all purposes, is always read-only anyway.
You may not see a case for actual corruptions, but my coworker
updated his tree on a box with 1.5.x, then tried to work on a box with
1.4.x (I think 1.4.2 back then), and ended up with a tree that was
unusable. He had to re-clone, and I think he got lucky recovering
pending changes (probably using 1.5.x on the branches with the changes,
as master was what got broken).
I still claim that there wasn't any corruptions.
Just for fun, just edit some document with Microsoft Office 95, then
open the same document with Office 2007 and save it with default
settings. Now try to open it back with Office 95. It won't work.
Does that mean that the document got corrupted?
My point is not that change is always bad, but that we should
really look forward to what we're doing, and that "it's in the release
notes" is not sufficient if an older git gives an incomprehensible error
or a silent problem. I was responding to the cavalier statement "well,
it's in the release notes, so don't worry about older versions".
Your allegation of corruptions is cavalier just as well.
I'm telling you that there won't be any such corruption. Just like in
the M$ Office case, it is expected that newer versions make data
unusable by older versions at some point -- that's the inevitable side
effect of progress.
And we cannot always anticipate what kind of incompatibility will be
worth making in the future, so it is hard to come with proper error
messages in all cases today.
Recent enough Git versions do suggest upgrading when they refuse to
access repositories though, and later Git versions can be configured to
remain compatible with old Git versions. And we also document it.
And when you still cannot figure it out on your own, then there is that
free support available on a public mailing list, and even an IRC
channel.
So I don't see how we could do better in that regard. Carving the
repository format in stone to keep ancient versions working forever is
_not_ a solution.
Nicolas
From: Joel Becker <hidden> Date: 2016-06-15 22:44:00
On Mon, Dec 17, 2007 at 03:41:24PM -0500, Nicolas Pitre wrote:
On Mon, 17 Dec 2007, Joel Becker wrote:
quoted
You may not see a case for actual corruptions, but my coworker
updated his tree on a box with 1.5.x, then tried to work on a box with
1.4.x (I think 1.4.2 back then), and ended up with a tree that was
unusable. He had to re-clone, and I think he got lucky recovering
pending changes (probably using 1.5.x on the branches with the changes,
as master was what got broken).
I still claim that there wasn't any corruptions.
Just for fun, just edit some document with Microsoft Office 95, then
open the same document with Office 2007 and save it with default
settings. Now try to open it back with Office 95. It won't work.
Does that mean that the document got corrupted?
No, but when you try to re-open it with Office 2007, you expect
it to work, don't you? His master was messed up even for 1.5.x. It was
now months ago, so I don't quite remember all the details, but I think
you'd agree that "1.5.x no longer works" is not correct.
I'm telling you that there won't be any such corruption. Just like in
the M$ Office case, it is expected that newer versions make data
unusable by older versions at some point -- that's the inevitable side
effect of progress.
Sure, we're not complaining about that. We complain some about
the fast pace (at the time he had his problem, 1.4 installs were not
unusual, and Junio's response suggested that "I use NFS" wasn't strongly
considered as a use case), but more we complain about the obscurity of
the reason. If it's obvious what happened (not the specifics, just
"please upgrade" or "repository format changed" or something), the user
moves along.
And we cannot always anticipate what kind of incompatibility will be
worth making in the future, so it is hard to come with proper error
messages in all cases today.
How hard is it? We have core.repositoryformatversion. We
undoubtably have headers on our files. As an example, an older version
should be able to ascertain 1) this is a pack file 2) I don't know how
to read it. Thus, it should always be able to tell the user as such.
This is different from reporting "invalid pack file" or "corrupt pack
file", or "garbage in tree". Filesystems, as an example, set
compatibility bits or version levels. When an old kernel tries to mount
it, it does not say "corrupt filesystem", it says "this filesystem has a
feature I don't understand, I'm going to be nice and not do anything,
please upgrade". This is clear, even though the older kernel doesn't
have any specifics about what the new feature is.
So I don't see how we could do better in that regard. Carving the
repository format in stone to keep ancient versions working forever is
_not_ a solution.
Once again, we're not asking for that. We're asking that you
think ahead to what can change, and plan for it, so you can tell the
user. If the user has a clear idea where to go next, the can solve the
rest themselves.
Look, not everyone reads this mailing list. No one outside of
this list reads the Release Notes. They get their upgrade via yum or
apt-get, along with 100 other packages. You can't assume that 3 months
of feature discussion here is going to be known to your average user.
Joel
--
"Vote early and vote often."
- Al Capone
Joel Becker
Principal Software Developer
Oracle
E-mail: joel.becker@oracle.com
Phone: (650) 506-8127
From: J. Bruce Fields <hidden> Date: 2016-06-15 22:44:00
On Mon, Dec 17, 2007 at 01:13:18PM -0800, Joel Becker wrote:
Sure, we're not complaining about that. We complain some about
the fast pace (at the time he had his problem, 1.4 installs were not
unusual, and Junio's response suggested that "I use NFS" wasn't strongly
considered as a use case), but more we complain about the obscurity of
the reason. If it's obvious what happened (not the specifics, just
"please upgrade" or "repository format changed" or something), the user
moves along.
By the way, just as a data point: I do keep some git repositories on
NFS, and access them from multiple machines with different git versions
(not on purpose--it's just that the machines don't all run the same
distro, so it'd be extra work to give them all the same version). I
don't use anything older than 1.5.0. If the repository became unusable
on one of those machines without warning it'd be annoying.
---b.
From: Nicolas Pitre <hidden> Date: 2016-06-15 22:44:00
On Mon, 17 Dec 2007, J. Bruce Fields wrote:
By the way, just as a data point: I do keep some git repositories on
NFS, and access them from multiple machines with different git versions
(not on purpose--it's just that the machines don't all run the same
distro, so it'd be extra work to give them all the same version). I
don't use anything older than 1.5.0. If the repository became unusable
on one of those machines without warning it'd be annoying.
What the v1.5.5 release notes will say is that you'll have to set
pack.indexversion=1 to remain compatible with pre-1.5.2 Git versions.
And even if you forget about it then there'll be a simple way to regain
compatibility after the facts.
Nicolas
From: J. Bruce Fields <hidden> Date: 2016-06-15 22:44:00
On Mon, Dec 17, 2007 at 04:52:16PM -0500, Nicolas Pitre wrote:
On Mon, 17 Dec 2007, J. Bruce Fields wrote:
quoted
By the way, just as a data point: I do keep some git repositories on
NFS, and access them from multiple machines with different git versions
(not on purpose--it's just that the machines don't all run the same
distro, so it'd be extra work to give them all the same version). I
don't use anything older than 1.5.0. If the repository became unusable
on one of those machines without warning it'd be annoying.
What the v1.5.5 release notes will say is that you'll have to set
pack.indexversion=1 to remain compatible with pre-1.5.2 Git versions.
Is there any reason not to make pack.indexversion=1 the default (for
preexisting repositories at the very least) and suggest in the release
notes that people set something else if they want the features the new
version provides?
--b.
And even if you forget about it then there'll be a simple way to regain
compatibility after the facts.
From: Nicolas Pitre <hidden> Date: 2016-06-15 22:44:00
On Mon, 17 Dec 2007, J. Bruce Fields wrote:
On Mon, Dec 17, 2007 at 04:52:16PM -0500, Nicolas Pitre wrote:
quoted
On Mon, 17 Dec 2007, J. Bruce Fields wrote:
quoted
By the way, just as a data point: I do keep some git repositories on
NFS, and access them from multiple machines with different git versions
(not on purpose--it's just that the machines don't all run the same
distro, so it'd be extra work to give them all the same version). I
don't use anything older than 1.5.0. If the repository became unusable
on one of those machines without warning it'd be annoying.
What the v1.5.5 release notes will say is that you'll have to set
pack.indexversion=1 to remain compatible with pre-1.5.2 Git versions.
Is there any reason not to make pack.indexversion=1 the default (for
preexisting repositories at the very least) and suggest in the release
notes that people set something else if they want the features the new
version provides?
That's already the case now.
But the thing is that index version 2 is better and actually plug a flaw
in the repacking process where unnoticed corruption could be repacked
otherwise. So for better repo integrity sake, it has to become the
default at some point.
Nicolas
From: Mark Fasheh <hidden> Date: 2016-06-15 22:44:00
Hi,
Just to "out" myself, I'm the "co-worker" whose name Joel has been
(politely) keeping anonymous.
On Mon, Dec 17, 2007 at 03:41:24PM -0500, Nicolas Pitre wrote:
quoted
You may not see a case for actual corruptions, but my coworker
updated his tree on a box with 1.5.x, then tried to work on a box with
1.4.x (I think 1.4.2 back then), and ended up with a tree that was
unusable. He had to re-clone, and I think he got lucky recovering
pending changes (probably using 1.5.x on the branches with the changes,
as master was what got broken).
I still claim that there wasn't any corruptions.
The following description is really vague because this was a while ago:
Something made my ocfs2.git tree unusable in that I could no longer do
common tasks, such as git-log, etc without getting messages about corrupted
refs.
I wish I had saved off some of the messages. Sorry.
I had to re-create my git tree several times before I learned by deduction
that it was the older versions of git on some of the machines that were
writing some sort of incompatible format.
Just for fun, just edit some document with Microsoft Office 95, then
open the same document with Office 2007 and save it with default
settings. Now try to open it back with Office 95. It won't work.
Does that mean that the document got corrupted?
Boy, I hope Microsoft Office isn't our bar for compatiblity here...
--Mark
--
Mark Fasheh
Senior Software Developer, Oracle
mark.fasheh@oracle.com
From: Nicolas Pitre <hidden> Date: 2016-06-15 22:44:00
On Mon, 17 Dec 2007, Mark Fasheh wrote:
Hi,
Just to "out" myself, I'm the "co-worker" whose name Joel has been
(politely) keeping anonymous.
On Mon, Dec 17, 2007 at 03:41:24PM -0500, Nicolas Pitre wrote:
quoted
quoted
You may not see a case for actual corruptions, but my coworker
updated his tree on a box with 1.5.x, then tried to work on a box with
1.4.x (I think 1.4.2 back then), and ended up with a tree that was
unusable. He had to re-clone, and I think he got lucky recovering
pending changes (probably using 1.5.x on the branches with the changes,
as master was what got broken).
I still claim that there wasn't any corruptions.
The following description is really vague because this was a while ago:
Something made my ocfs2.git tree unusable in that I could no longer do
common tasks, such as git-log, etc without getting messages about corrupted
refs.
I wish I had saved off some of the messages. Sorry.
I had to re-create my git tree several times before I learned by deduction
that it was the older versions of git on some of the machines that were
writing some sort of incompatible format.
Next time please don't hesitate to post your issue on this list. The
fix could have been so obvious to many people on the list, saving you
time and frustration. In your case I think the "fix" would have
consisted of simply running "git repack -a -d" on the machine with the
most recent Git version.
And if it was a case of real corruption then we certainly would have
liked to know about it ASAP.
quoted
Just for fun, just edit some document with Microsoft Office 95, then
open the same document with Office 2007 and save it with default
settings. Now try to open it back with Office 95. It won't work.
Does that mean that the document got corrupted?
Boy, I hope Microsoft Office isn't our bar for compatiblity here...
We can do better of course, like allowing you to still produce the old
format with a new version of Git. But sometimes format changes are
unavoidable for many good reasons.
Nicolas