Re: Separating generated files? (Re: Mercurial -> git)

Hi Christian,

The idea of using two separate repositories for source and generated
source is interesting. I would like to bring this to git mailing list,
they may provide insightul comments for your idea or even other
approaches.

Background for Git people: gambit-c was previously stored in
mercurial. The main source is in gambit-c (a Scheme implementation or
a Lisp dialect). *.scm files generate *.c, which will be compiled by
gcc as usual. Both *.scm and generated *.c are now stored in
mercurial. Gambit-C maintainers have recently decided to move to Git.

On 10/15/08, Christian Jaeger [off-list ref] wrote:

I wonder whether it would be a good idea, and good occasion to realize
 it, to move source files and generated files into separate repositories
 and 'link' those together using the git submodule feature.

 Expected advantages:

 - no clutter when looking through the history (can possibly be mitigated
 by constraining git log, git diff etc. to the non-generated paths only,
 although I don't think this is possible (cleanly) with the current
 directory structure); the same holds true for using "git format-patch"
 (one wouldn't usually want to include the generated files in diffs sent
 to the mailing list)

 - when merging branches, there will usually be no need to deal with
 merge conflicts in the generated files (one would just regenerate them
 instead)

 - [especially for files being generated not by Gambit itself (for
 example "configure"),] the files can be regenerated by differing
 [external] software versions without having to deal with those
 superfluous changes in the source repository.

 By still committing the generated files--to a different
 submodule--Gambit can still be updated through Git alone, and the
 possible advantage of tracking the generated files to see the effects of
 changes in compiler sources can still be had.

 Expected disadvantages:

 - all generated files need to reside in a separate directory structure;
 e.g. the file $BASEDIR/lib/_io.c would have to be at a place like
 $BASEDIR/build/lib/_io.c instead, where build/ is the submodule taking
 all generated files; since the "configure" file is expected to reside at
 the toplevel, I guess this would require that "make update" copies it
 from $BASEDIR/build/configure to $BASEDIR/configure (assuming that one
 cannot use a symlink because of portability reasons).

 - to commit the generated files, a separate step is necessary ("cd
 $otherrepo; git commit -a", or maybe easier create a "make
 commit_generated" make target?)

 - to make this work with the "source" repository residing at the
 toplevel, the Git superproject repository (of which the "source" and
 "build" repositories are submodules) would need to reside in a
 non-standard directory, like $BASEDIR/.gitsuperproject/ instead of the
 usual .git/, and using the GIT_DIR environment variable to access it,
 although this can probably be handled by make targets (i.e. "make
 update" would set GIT_DIR=$BASEDIR/.gitsuperproject when calling "git
 submodule update").

 - there may be some cases to flesh out; like, should "make update"
 really call "git submodule update" (which simply sets the submodules to
 the reference given by the superproject, throwing away changes done by
 the user in the submodules (they can be recovered from the git reflog,
 but may still be a surprise)) or should it run "git pull" in each
 submodule instead?

 I thought I'd bring this up now because if package maintainers need to
 adapt some things anyway, that may be a good time to do it now. (There's
 even the possibility to split the converted Mercurial repository into
 the source + build parts in retrospect now, which won't be possible
 anymore later on (without changing the sha1 sums of the whole Git
 history with the associated breakage of existing clones), although that
 may not be important.)

 I'm willing to help in the effort, although I don't know the build tools
 (autoconf and make) and their use in the setup well, so I would probably
 be quite a bit lost when doing it alone.

 Christian.

-- 
Duy