From: Yuval Adam <hidden> Date: 2016-06-15 22:53:26
As part of a public project to open-source the Israeli law code, we
are looking into ways of represent such data in a git repository.
The main challenge is to represent historical data _in a semantically
correct way_ within a git repository, while having the ability to
change data that has occurred in the past.
For example, we might have revisions B and C of a certain legal
document, commit to repo, and at a later time want to add revision A
to the proper place in the git commit tree (probably with rebasing or
replacing).
Allowing decentralization and updates is a major requirement.
We're trying to map out the various pros and cons of the different
options of maintaining such a repo.
Has anyone ever attempted something like this?
Are there any projects that build on the git plumbing which provide
wrapper APIs to handle historic data?
We really could use any reference or advice we can get on this subject.
Thanks,
--
Yuval Adam
http://y3xz.com
In message [off-list ref], Yuval Adam writes:
As part of a public project to open-source the Israeli law code, we
are looking into ways of represent such data in a git repository.
This is extremely cool. I wish others were forward thinking enough
to do this.
The main challenge is to represent historical data _in a semantically
correct way_ within a git repository, while having the ability to
change data that has occurred in the past.
Revision control shouldn't be used to change the past (even if git
allows this with sufficient amounts of pain/warning to all users).
What it is extremely good at is preserving the past and tracking the
changes that are made.
For example, we might have revisions B and C of a certain legal
document, commit to repo, and at a later time want to add revision A
to the proper place in the git commit tree (probably with rebasing or
replacing).
There is no problem doing this. I'll make up a mythical workflow
which might be realistic. Someone proposes a bill, so a branch for
the proposal is created. In many of the laws I am familiar with,
there is the text of the law and then the text says "Amend V.5.12.A.b
to add '25: or to commit a nasal offense (as defined in V.5.12.A) with
a shoe'". So the branch might contain the text of the proposed law
and then actually go through to the document V.5.12.A.b and add the
new data to the appropriate file (in an ideal world that might be an
automatic process, but laws are rarely so precise). The proposed law
changes and the bill text changes would be committed onto the branch.
As the bill goes through committee people make changes, adding things,
removing things, etc. Each change is a commit. One example change
might be a new change saying "remove the change made 2 days ago" or
"make the current version the version from 10 days ago". Both of
those specific changes would ideally be positive changes. You would
not actually be deleting the change made two days ago or removing all
changes made between 10 days ago and now, you would be making a new
commit to remove the effects of the unwanted changes.
When the negotiations are over and assuming the bill gets all three
readings (each reading could be a "tag" to document exactly what was
read) and voted into a law, you would then merge the bill branch into
the "law" branch which represents the actual legally active laws.
This could be done as a "squash" merge which hides all of the
committee negotiations or it could be done as a normal merge which
allows the history of the negotiations to be visible, or, depending on
the visibility of the committee negotiations, you could even do a
combination of the two.
And yes, git supports more complex processes automatically, like each
Knesset member making their own proposed changes and the committee
chair merging the appropriate version in if it was approved and the
others being either discarded or archived for history but not
incorporated.
Allowing decentralization and updates is a major requirement.
git is extremely good at this.
We're trying to map out the various pros and cons of the different
options of maintaining such a repo.
Ideally the data being represented would be structured, textual, and
somewhat line oriented, plain text/UTF-8 files (no matter the word
direction) like this email are ideal. Committing binary Office
documents (Word, OpenXML, ODF, etc) is not ideal, since under most
circumstances/without a lot of work you are not going to get good
differences so that you can easily see the history of the law. You
can write custom binary drivers to extract this difference information
from these binary documents, but that is the "lot of work" I was
talking about.
You additionally might want to have separate repositories for separate
groups of laws to prevent repositories from getting unwieldy. There
are tools which let you group repositories together.
Has anyone ever attempted something like this?
Many people use git to track living documents. Perhaps not law per
se, but I don't particularly see why that would matter.
Are there any projects that build on the git plumbing which provide
wrapper APIs to handle historic data?
Are you talking about "get rid of that change, it was bad" and
"restore this version of the document as the good one" or "how do I
import 64 years of law into git"? Git provides native tools to handle
both.
We really could use any reference or advice we can get on this subject.
I'll point you at http://progit.org/book/ as a general reference about
git and http://sethrobertson.github.com/GitBestPractices/ as a
reference about best practices.
-Seth Robertson
From: Yuval Adam <hidden> Date: 2016-06-15 22:53:26
On Fri, Mar 30, 2012 at 6:10 PM, Seth Robertson [off-list ref] wrote:
Revision control shouldn't be used to change the past (even if git
allows this with sufficient amounts of pain/warning to all users).
What it is extremely good at is preserving the past and tracking the
changes that are made.
This is exactly what we _do_ want to do.
Our use case for this is like so:
"ok, this is how the law is today, and we're not quite sure how it got
to this point"
But then some X time later:
"so we found out that clauses (1), (e) and (X) were changed on March
30, 1957, and we want to know this for future reference"
So, yes, we do need a way of knowing (blaming?) what happened in the
past and how the law was shaped over time.
Is this something that is definitively complicated with git?
--
Yuval Adam
http://y3xz.com
In message [off-list ref], Yuval Adam writes:
On Fri, Mar 30, 2012 at 6:10 PM, Seth Robertson [off-list ref] wrote:
> Revision control shouldn't be used to change the past (even if git
> allows this with sufficient amounts of pain/warning to all users).
> What it is extremely good at is preserving the past and tracking the
> changes that are made.
This is exactly what we _do_ want to do.
Is this something that is definitively complicated with git?
Ah, I understand now. I imagine others will chime in as well, but
this should not be too complex with git. You can easily go back into
history and change it. The problem comes in when you have shared your
repository with other people.
In general, rewriting public history is a bad idea because git cannot
tell the difference between someone adding to history for good reasons
(expanding on known history) and bad reasons (retroactively rewriting
the law to add a loophole).
You can absolutely do it, but then you have to "force push" your
changes to the master server to override the history (assuming that is
allowed, and it typically is not by default) and then everyone else
would have to do special things (`git pull --rebase` in the simple
case, rebuilding branches and tags in more complex cases) to get the
new history. Clearly for something like the law and the probable
complex workflow it will have, this isn't a good method.
What I would probably suggest is having either a historical branch or
a historical repository which is allowed and expected to be rewritten.
The changes would then be confined to places where active
"development" would not be occurring and the process to recover from
the retroactive changes could be automated. The "git replace" and
"git grafts" (the last might be deprecated) functionality could be
used to merge the two histories together so it is transparent to those
who need a consistent view from now to the beginning. With a separate
repo then the normal users who only care about the recent changes and
current state don't ever have to do anything special or worry about
the history changes, but it should work in either case.
-Seth Robertson
From: Jakub Narebski <hidden> Date: 2016-06-15 22:53:26
Seth Robertson [off-list ref] writes:
In message [off-list ref], Yuval Adam writes:
On Fri, Mar 30, 2012 at 6:10 PM, Seth Robertson [off-list ref] wrote:
> Revision control shouldn't be used to change the past (even if git
> allows this with sufficient amounts of pain/warning to all users).
> What it is extremely good at is preserving the past and tracking the
> changes that are made.
This is exactly what we _do_ want to do.
Is this something that is definitively complicated with git?
Ah, I understand now. I imagine others will chime in as well, but
this should not be too complex with git. You can easily go back into
history and change it. The problem comes in when you have shared your
repository with other people.
In general, rewriting public history is a bad idea because git cannot
tell the difference between someone adding to history for good reasons
(expanding on known history) and bad reasons (retroactively rewriting
the law to add a loophole).
You can absolutely do it,
For example using `git filter-branch`, or grafts mechanism plus said
git-filter-branch, or interactive rebase for changes closer to current
version, or `git commit --amend` for latest version (latest commit).
but then you have to "force push" your
changes to the master server to override the history (assuming that is
allowed, and it typically is not by default) and then everyone else
would have to do special things (`git pull --rebase` in the simple
case, rebuilding branches and tags in more complex cases) to get the
new history. Clearly for something like the law and the probable
complex workflow it will have, this isn't a good method.
Well, if nobody is basing their work on this repository, and it is
meant as read-only source of information, that doesn't matter much.
What I would probably suggest is having either a historical branch or
a historical repository which is allowed and expected to be rewritten.
[...]
Yet another solution would be to fix mistakes using `git replace`
mechanism. It doesn't as much rewrite history, as paste on fixes;
this of course requires setting up sharing of those replacements
(fixes).
See git-replace(1) manpage for more information.
--
Jakub Narebski
From: Markus Elfring <hidden> Date: 2016-06-15 22:53:27
Our use case for this is like so:
"ok, this is how the law is today, and we're not quite sure how it got
to this point"
But then some X time later:
"so we found out that clauses (1), (e) and (X) were changed on March
30, 1957, and we want to know this for future reference"
I imagine that technical challenges come from a different view for your use
case. Content management systems can eventually show differences for
line-oriented text files easily. But I guess that you are also interested in the
maintenance of higher level semantic data structures that are usually contained
in outlines.
How would you like to build relationships between commit logs and changes to
items like chapters, sections, paragraphs and sentences?
Do you need to combine several information sources to generate a document query
and result representation you desire?
Regards,
Markus