Hi Christian,
The idea of using two separate repositories for source and generated
source is interesting. I would like to bring this to git mailing list,
they may provide insightul comments for your idea or even other
approaches.
I think the first question is: do you (and why) need to use a version
control system for generated files?
--
Matthieu
From: Michael J Gruber <hidden> Date: 2016-06-15 22:45:29
Matthieu Moy venit, vidit, dixit 15.10.2008 17:30:
"Nguyen Thai Ngoc Duy" [off-list ref] writes:
quoted
Hi Christian,
The idea of using two separate repositories for source and generated
source is interesting. I would like to bring this to git mailing list,
they may provide insightul comments for your idea or even other
approaches.
I think the first question is: do you (and why) need to use a version
control system for generated files?
I guess we can take "yes" for granted for the first part ;)
As for the why: In cases like this one it's interesting to compare
(read: diff) the output generated by different versions of the input.
I wonder whether a clever use of "excludes" and GIT_DIR would allow
tracking the different filesets in the same dir, but using different
repos. I'm just afraid it's a fragile setup, in the sense that it relies
on config stuff which is not tracked (and thus not reproduced
automatically on clone).
Michael
From: Christian Jaeger <hidden> Date: 2016-06-15 22:45:29
Matthieu Moy wrote:
I think the first question is: do you (and why) need to use a version
control system for generated files?
The project in question is a self-hosting compiler which compiles to C
as an intermediary language. Providing the generated C files to users
makes installation easy (it avoids the bootstrapping issue). So it's
more 'severe' of an issue than just one of for example generating
documentation files using a 3rd-party tool.
What may make matters worse, is that there are interdependencies between
a number of hand-written C files and the generated files, so it's not
always possible to use an older compiler version to reproduce the
generated C files for a newer compiler; so if you want to merge newer
compiler sources, you may also need the generated files, at least if you
want that without fuss. So, there is always a need to somehow transmit
the generated files too. I guess that this is easier than code the
system in a way to always allow backwards compatibility (I haven't
worked on the compiler itself yet, so this is a guess and may need
confirmation).
Apart from that, I've found it useful (in another project, writing a
document translator) to keep generated files in a VCS (Git) as well (I
checked them into the *same* repository as the translator source, even
if it felt ugly (for the previously mentioned reasons)), as then when I
changed the translator, I could easily see where it had effect on the
generated output. It can even serve as a debugging help kind of like a
test suite does. This may be the case here, too (again, I'm guessing here).
How are other compiler projects which are bootstrapping via C dealing
with this?
Christian.
From: Christian Jaeger <hidden> Date: 2016-06-15 22:45:29
Michael J Gruber wrote:
I wonder whether a clever use of "excludes" and GIT_DIR would allow
tracking the different filesets in the same dir, but using different
repos. I'm just afraid it's a fragile setup, in the sense that it relies
on config stuff which is not tracked (and thus not reproduced
automatically on clone).
I expect that using a superproject repository to tie together the two
repositories is good and necessary because it is the link that allows to
specify which commit in the repo of generated files belongs together
with a commit in the repo of source files. So just using two separate
repositories without making them submodules of a superproject does not
seem to be a good idea to me.
Once there is a superproject repository, one could also commit config
files of the submodules into it (I'm not sure what that will buy
though--.gitignore is outside and can committed anyway, at least as long
as not both repositories are overlaid as you suggest).
You're probably right that strictly speaking, there is no need to move
generated files out into a separate directory tree; but I think doing
the move would be worthwhile since it takes away one level of complexity
(you can then access the build/.git repository without the need of
setting GIT_DIR), and because it may be a good idea anyway (for example
it will be easier to grep the sources without getting hits from the
generated files). [Also, the exclude patterns wouldn't be easy, as we
couldn't really just exclude all *.c files from the view of the source
repository, since there are also some hand-crafted ones; the excludes
would need full paths which would have to be kept up to date manually,
unless we wanted to live with the fact that newly created manual .c
files would be added using "git add -f".]
Christian.
From: Christian Jaeger <hidden> Date: 2016-06-15 22:45:29
I wrote:
Michael J Gruber wrote:
quoted
I wonder whether a clever use of "excludes" and GIT_DIR would allow
tracking the different filesets in the same dir, but using different
repos. I'm just afraid it's a fragile setup, in the sense that it relies
on config stuff which is not tracked (and thus not reproduced
automatically on clone).
I expect that using a superproject repository to tie together the two
repositories is good and necessary because it is the link that allows
to specify which commit in the repo of generated files belongs
together with a commit in the repo of source files. So just using two
separate repositories without making them submodules of a superproject
does not seem to be a good idea to me.
(In the meantime I've read the following pages:
http://nopugs.com/2008/09/06/ext-tutorialhttp://nopugs.com/2008/09/04/why-exthttp://flavoriffic.blogspot.com/2008/05/managing-git-submodules-with-gitrake.html
(a post to the latter article suggests to use subtree merging instead,
but that would be a very bad match for our use case; the mentioned
problem of merging of the git superproject makes me think, though--the
superproject could be updated only by the one person doing the publish
onto the public repository, but then it leaves the problem of handling
merges by developers completely unsolved.)
)
I'm starting to think that maybe a better idea than the
superproject+2submodules approach would be just using the two
repositories ("source" + "build"), and storing the linking information
inside the "build" repository (by adding the source repository commitid
to every commit message in the build repository [or using tags, but that
doesn't seem a better idea]), and use a program that is able to check
out the matching "build" repository for a given "source" repository
checkout.
I'm willing to write this program (let's call it
"intergit-find-matching-commit-in" for the purpose of this email);
question: which language to write it in, is Perl good? (C would be a
hassle for Windows users because of the C compiler requirement; shell
may be too limited.)
Description of the workings in more detail:
- one would work with the "source" repository just as one would with any
project only employing one repository; do some changes to the project,
commit them, test them (includes regeneration of generated files);
- once in a while one would commit the current generated files in the
"build" repository; by either (a) using a make target (like "make
commit_generated") which runs something like
eval "cd build; git commit -m 'generated files for source repository
commit `git rev-parse HEAD`'"
or (b) setting up a build/.git/hooks/commit-msg script which appends
'generated files for source repository commit `git rev-parse HEAD`' line
to the commit message given from running "cd build; git commit -a" manually.
- for publication, one would push both the "source" as well as the
"build" repository (i.e. "cd build; git push; cd ..; git push")
- for checkout (our "make update" make target), about the following
would happen:
git pull
eval "(cd build; git checkout `intergit-find-matching-commit-in build`)"
where "intergit-find-matching-commit-in build" would first refresh an
index of the links (iterate over all unseen commits, parse commit
messages for /source repository commit (\w+)/ and store $1 =>
$commitid_in_build_repo mappings in the index), then go through "git log
--pretty=format:%H" (should I also specify --topo-order (or
--date-order)?) looking up the commitids in the index, stopping at the
first match and outputting the mapped $commitid_in_build_repo.
This way, the "latest" or "probably best-matching" corresponding commit
in the "build" repo can always be found, even if the "source" repo is
ahead, which should allow building the compiler even if none is
previously installed. This workflow seems more natural than the
superproject+submodules approach, and it seems to entail no hassle with
merge issues (only the "source" repo really needs proper merging;
merging the "build" repo would only be worthwhile for maintaining the
history, and as mentioned if there are conflicts, one would probably
usually just regenerate the files there; there's no need to maintain
linking info (with associated merge etc issues) in a separate entity
(superproject) anymore, and during development, commits to the "build"
repo need only be done if backwards-incompatible changes have been
introduced).
Does anyone else think this is sane/interesting? Should I go ahead
implementing this? Any comments, like on how the interface of the
intergit-find-matching-commit-in tool should look like?
Christian.
On Thu, Oct 16, 2008 at 2:00 PM, Christian Jaeger
[off-list ref] wrote:
(In the meantime I've read the following pages:
http://nopugs.com/2008/09/06/ext-tutorialhttp://nopugs.com/2008/09/04/why-exthttp://flavoriffic.blogspot.com/2008/05/managing-git-submodules-with-gitrake.html
(a post to the latter article suggests to use subtree merging instead, but
that would be a very bad match for our use case; the mentioned problem of
merging of the git superproject makes me think, though--the superproject
could be updated only by the one person doing the publish onto the public
repository, but then it leaves the problem of handling merges by developers
completely unsolved.)
)
I'm starting to think that maybe a better idea than the
superproject+2submodules approach would be just using the two repositories
("source" + "build"), and storing the linking information inside the "build"
repository (by adding the source repository commitid to every commit message
in the build repository [or using tags, but that doesn't seem a better
idea]), and use a program that is able to check out the matching "build"
repository for a given "source" repository checkout.
I'm willing to write this program (let's call it
"intergit-find-matching-commit-in" for the purpose of this email); question:
which language to write it in, is Perl good? (C would be a hassle for
Windows users because of the C compiler requirement; shell may be too
limited.)
Description of the workings in more detail:
- one would work with the "source" repository just as one would with any
project only employing one repository; do some changes to the project,
commit them, test them (includes regeneration of generated files);
- once in a while one would commit the current generated files in the
"build" repository; by either (a) using a make target (like "make
commit_generated") which runs something like
eval "cd build; git commit -m 'generated files for source repository commit
`git rev-parse HEAD`'"
or (b) setting up a build/.git/hooks/commit-msg script which appends
'generated files for source repository commit `git rev-parse HEAD`' line to
the commit message given from running "cd build; git commit -a" manually.
- for publication, one would push both the "source" as well as the "build"
repository (i.e. "cd build; git push; cd ..; git push")
- for checkout (our "make update" make target), about the following would
happen:
git pull
eval "(cd build; git checkout `intergit-find-matching-commit-in build`)"
where "intergit-find-matching-commit-in build" would first refresh an index
of the links (iterate over all unseen commits, parse commit messages for
/source repository commit (\w+)/ and store $1 => $commitid_in_build_repo
mappings in the index), then go through "git log --pretty=format:%H" (should
I also specify --topo-order (or --date-order)?) looking up the commitids in
the index, stopping at the first match and outputting the mapped
$commitid_in_build_repo.
This way, the "latest" or "probably best-matching" corresponding commit in
the "build" repo can always be found, even if the "source" repo is ahead,
which should allow building the compiler even if none is previously
installed. This workflow seems more natural than the superproject+submodules
approach, and it seems to entail no hassle with merge issues (only the
"source" repo really needs proper merging; merging the "build" repo would
only be worthwhile for maintaining the history, and as mentioned if there
are conflicts, one would probably usually just regenerate the files there;
there's no need to maintain linking info (with associated merge etc issues)
in a separate entity (superproject) anymore, and during development, commits
to the "build" repo need only be done if backwards-incompatible changes have
been introduced).
Does anyone else think this is sane/interesting? Should I go ahead
implementing this? Any comments, like on how the interface of the
intergit-find-matching-commit-in tool should look like?
This script only generates the html / man branches, it doesn't help find
the right version for a given git version, right?
The differences are:
- the html / man branches have a strictly linear history and are
centrally maintained. This solves the distribution issue for end users.
But while developping the compiler, the developers may need to go back
in the history of their own development (e.g. when the current compiler
doesn't work anymore), and the suspected usefulness of being able to see
and track differences in the generated code also isn't available for a
strictly central approach.
- the script above is only for creating and committing the derived
files, in a hook similar to the one I suggested in
build/.git/hooks/commit-msg; this is the "cd build; git commit -m
'generated files for source repository commit
> `git rev-parse HEAD`'" part; the more interesting part comes from
automatically finding the right commit in the generated branches for a
given source commit. This is what I intend to solve with the
"intergit-find-matching-commit-in" script. Said in a simpler way: the
git html / man branches do not offer automatically resolvable linking.
Christian.
This script only generates the html / man branches, it doesn't help find the
right version for a given git version, right?
Right, one script to generate and one to get the right version.
The differences are:
- the html / man branches have a strictly linear history
Yes, because in this case it is not needed to replicate the whole
history, but it could be improved.
and are centrally maintained. This solves the distribution issue for end users. But while
developping the compiler, the developers may need to go back in the history
of their own development (e.g. when the current compiler doesn't work
anymore), and the suspected usefulness of being able to see and track
differences in the generated code also isn't available for a strictly
central approach.
So, you can divide the problem in two: (a) generated files in the
remote repositories (these can be generated automatically on the
server or in a dedicated server) (b) local generated files for local
commits. If both follow the same format to specify the original commit
you can use the same script to get it.
- the script above is only for creating and committing the derived files, in
a hook similar to the one I suggested in build/.git/hooks/commit-msg; this
is the "cd build; git commit -m 'generated files for source repository
commit
quoted
`git rev-parse HEAD`'" part; the more interesting part comes from
automatically finding the right commit in the generated branches for a given
source commit. This is what I intend to solve with the
"intergit-find-matching-commit-in" script. Said in a simpler way: the git
html / man branches do not offer automatically resolvable linking.
They offer this (Autogenerated HTML docs for v1.6.0.2-530-g67faa) but
there is no script around it.
My point was that there are other project keeping generated files (and
sometimes I would like it too), so you can see what they are doing. At
the end, maybe, you system could be usefull for them also.
Santi