Re: Maintaining historical data in a git repo

Yuval Adam [off-list ref] writes:

Is this something that is definitively complicated with git?

That's not really "is it complicated with git" question, I would have to
say.  Any version control system you would build history starting from one
point going _forward_, never inserting past event as you dig back.

Surely, you could fake it by rewriting history, but I do not think SCM is
particularly geared towards such a use case *while* investigating the
history of the law and recording your findings.

I do agree that once such a discovered history is *complete*, it would be
nice to record it in a SCM with a powerful history digging capability in
chronological order, though.

On Fri, Mar 30, 2012 at 7:52 PM, Junio C Hamano [off-list ref] wrote:

That's not really "is it complicated with git" question, I would have to
say.  Any version control system you would build history starting from one
point going _forward_, never inserting past event as you dig back.

That is true.
It is very clear to us that an SCM is optimized for the prevalent use
case, which is tracking code (well, mostly code) as it is written.
Naturally this always starts at some point in time and progresses into
the future.

However, we perceive git as a very powerful tool, that can fit
beautifully with the way legislation works today.
The challenge for us - should we choose to accept it ;) - is to build
a set of wrapper tools that allow us to use git in such a way, while
enabling us to build up past history.

Yes, this is not the usual use case, but we're highly motivated on
making this work.
We believe this could also be an interesting experience for the git
community in seeing how the git plumbing can be used for other cases,
even if they veer off on some weird tangent.

We'll definitely be back with more questions and updates, as we progress.
Thanks, everyone, for your responses and feedback!

--
Yuval Adam
http://y3xz.com

On Fri, 30 Mar 2012, Yuval Adam wrote:

On Fri, Mar 30, 2012 at 7:52 PM, Junio C Hamano [off-list ref] wrote:

quoted

That's not really "is it complicated with git" question, I would have to
say.  Any version control system you would build history starting from one
point going _forward_, never inserting past event as you dig back.

That is true.
It is very clear to us that an SCM is optimized for the prevalent use
case, which is tracking code (well, mostly code) as it is written.
Naturally this always starts at some point in time and progresses into
the future.

However, we perceive git as a very powerful tool, that can fit
beautifully with the way legislation works today.
The challenge for us - should we choose to accept it ;) - is to build
a set of wrapper tools that allow us to use git in such a way, while
enabling us to build up past history.

Yes, this is not the usual use case, but we're highly motivated on
making this work.
We believe this could also be an interesting experience for the git
community in seeing how the git plumbing can be used for other cases,
even if they veer off on some weird tangent.

We'll definitely be back with more questions and updates, as we progress.
Thanks, everyone, for your responses and feedback!

you may want to take a hint from how the linux repository works.

When git was created, the as-of-then current version was commited as the 
base and development went on from there.

Later on the linux historical repository was created (and re-created over 
time as other versions were found).

The git graft command can be used to join the 'current' repository to the 
'historical' repository so that they can be treated as one.

I strongly suspect that something along these lines is what you are 
needing.

David Lang

On Fri, Mar 30, 2012 at 4:39 PM, Yuval Adam [off-list ref] wrote:

However, we perceive git as a very powerful tool, that can fit
beautifully with the way legislation works today.
The challenge for us - should we choose to accept it ;) - is to build
a set of wrapper tools that allow us to use git in such a way, while
enabling us to build up past history.

If you're willing to put some time into either writing new tools or
doing complicated work by hand, you could use git to keep track of the
history's history.  Have two branches: a real "master" branch and a
"meta" branch to keep track of master's history.  The former is what
end users would see: the most accurate history of the code to date.
The latter is what "developers" would use to rebuild the master branch
with new information (say, adding A before B and C).

To do this, you could try the following: Use normal git commands on
the master branch, but every time you change master (say, commit or
rebase), also make a special commit on the meta branch with the first
parent being a reference to the new value of master.  Use the
remaining parents as "normal" references to previous meta commits, and
use an empty tree.  Now, the meta branch contains a complete history
of the history, though viewing it will be extremely ugly unless you
develop a custom tool to deal with its special form.

Optionally, on the server, you could set up an update hook to disallow
updates of the master branch and disallow non-fast-forward updates of
the meta branch, and a post-receive hook to the master branch to point
to the first parent of the meta branch each time the meta branch is
updated.

One caveat is that you must be careful about merges on the meta
branch, since git's default strategy will automatically do the wrong
thing.  You could write your own merge strategy to handle this.
(Sadly there does not appear to be a way to use this strategy
automatically on per-branch basis.)  Another workaround would be to
use something that is unmergable in the tree of the meta commit,
rather than an empty tree - say, a single file with the commit ID of
the master branch - which would prevent the default strategy from
trivially and incorrectly merging.

Using such a system would be awkward by hand but not terribly
difficult to automate.  You could create a "git-meta-commit" command
to create a meta commit for the current branch.  You might find
contrib/examples/git-merge.sh useful as a guide for how to do this.
If you'd like more details, please ask.

It would be nice if you could write a hook that automatically creates
a meta commit every time master's reflog is updated, but this does not
seem possible at the moment.

Mark Lodato wrote:
On Fri, Mar 30, 2012 at 4:39 PM, Yuval Adam [off-list ref] wrote:

quoted

However, we perceive git as a very powerful tool, that can fit
beautifully with the way legislation works today.
The challenge for us - should we choose to accept it ;) - is to build
a set of wrapper tools that allow us to use git in such a way, while
enabling us to build up past history.

If you're willing to put some time into either writing new tools or
doing complicated work by hand, you could use git to keep track of the
history's history.  Have two branches: a real "master" branch and a
"meta" branch to keep track of master's history.  The former is what
end users would see: the most accurate history of the code to date.
The latter is what "developers" would use to rebuild the master branch
with new information (say, adding A before B and C).

Why not just skip the master branch altogether? Create a branch named for today's date and commit to it the history of the law as seen at today. When historic changes are discovered, create a branch where it fits into the record (named for the date of the discovery), commit the new version, then cherry pick the remainder of the history from then on top of it. Ending up with two parallel historic records showing what you thought the history of the document was up until last Wednesday and the new branch of what we know now. Having the same version of the document in multiple branches has no storage penalties in git.

On Fri, Mar 30, 2012 at 22:39, Yuval Adam [off-list ref] wrote:

However, we perceive git as a very powerful tool, that can fit
beautifully with the way legislation works today.
The challenge for us - should we choose to accept it ;) - is to build
a set of wrapper tools that allow us to use git in such a way, while
enabling us to build up past history.

You can always solve this by having two repositories, you have one
canonical Git repository with your laws using some text-based format
to describe when each change was added.

You'd never rewrite the history of this repository since it would
represent the history of your project to give a commit timeline to the
law, and not attempt to make your commit log reflect changes in the
law.

You could then have tools to export another Git history from that
original repository, that one would be constantly rewritten and nobody
would base changes on that.

You could also make the two one and the same, but you don't have to.