Looking for a way to set up Git correctly

I have a situation.

I have started a web project (call it branch1), and have maintained it 
without a version control system for quite some time.
Then, I copied it to another folder (branch2) and while the project remained 
essentially the same, I have changed a few of internal paths and some 
variable names inside the files.
Then, a few months later on, I copied branch2 to a folder called branch3 and 
also modified some of the variable names and some of the internal structure 
of the files.

Thus I ended up with 3 folders on my local HDD with pretty much the same 
file names and folder structure and everything, and most of the file 
content, except those small deltas that made those files different for each 
branch.

I guess it's never too late, and now I want to put these 3 projects into a 
version control system, and I chose git.

Now, this can be either really simple or really complicated.  My first 
question is:  how do I set the repository up in the proper way where I could 
work on all 3 projects separately, with additional possibility of working on 
branch1 only and later committing my changes to branch2 and branch3.  (Since 
projects are virtually identical, a fix in one branch usually needs to be 
propagated to other branches)
First, I assume I will use a single repository for this.  Then, do I simply 
set up 3 branches and start using them, or is there a way to set git up to 
capitalize on the projects being nearly identical?

My second question is that each branch has a huge folder with image data. 
By huge I mean 1 to 4Gb, depending on the branch.  Since images are not 
directly relevant to the development work, is there a way to not include 
those folders in git?  To be honest though, I probably should include them, 
but I wanted to ask about this separately as git repository may be get 
large, since all 3 branches may grow to 9Gb or so.

Thus I am looking for a git way to handle my situation.  Is this simple or 
is is hard?
Are there any recommendations before I jump in?
Dennis 

On Thu, Nov 11, 2010 at 04:25, Dennis [off-list ref] wrote:

I have started a web project (call it branch1), and have maintained it
without a version control system for quite some time.
Then, I copied it to another folder (branch2) and while the project remained
essentially the same, I have changed a few of internal paths and some
variable names inside the files.
Then, a few months later on, I copied branch2 to a folder called branch3 and
also modified some of the variable names and some of the internal structure
of the files.

Thus I ended up with 3 folders on my local HDD with pretty much the same
file names and folder structure and everything, and most of the file
content, except those small deltas that made those files different for each
branch.

I guess it's never too late, and now I want to put these 3 projects into a
version control system, and I chose git.

Now, this can be either really simple or really complicated.  My first
question is:  how do I set the repository up in the proper way where I could
work on all 3 projects separately, with additional possibility of working on
branch1 only and later committing my changes to branch2 and branch3.  (Since
projects are virtually identical, a fix in one branch usually needs to be
propagated to other branches)
First, I assume I will use a single repository for this.  Then, do I simply
set up 3 branches and start using them, or is there a way to set git up to
capitalize on the projects being nearly identical?

Assuming I've got the relationships of your "branches" right:

$ cp -a branch1 branch && cd branch
$ git init
$ echo /huge-images/ >.gitignore
$ git add .gitignore; git add .; git commit; git branch branch1
$ git checkout -b branch2
$ cp -a ../branch2 .
$ git add .; git commit
$ git checkout -b branch3
$ cp -a ../branch3 .
$ git add .; git commit

My second question is that each branch has a huge folder with image data. By
huge I mean 1 to 4Gb, depending on the branch.  Since images are not
directly relevant to the development work, is there a way to not include
those folders in git?  To be honest though, I probably should include them,
but I wanted to ask about this separately as git repository may be get
large, since all 3 branches may grow to 9Gb or so.

Thus I am looking for a git way to handle my situation.  Is this simple or
is is hard?

If you add the images you will eventually run into problems (heavy
swapping, for one).
Git is not really setup to work with big binary files (a file must fit into
memory completely).

* Dennis [off-list ref] wrote:

Hi,

Now, this can be either really simple or really complicated.  My first 
question is:  how do I set the repository up in the proper way where I 
could work on all 3 projects separately, with additional possibility of 
working on branch1 only and later committing my changes to branch2 and 
branch3.  

As first step you could create 3 separate git repos in each directory
and add everything to it (git init, git add -A, git commit). Then 
rename the branches properly (so instead of "master", they'll be called
"branch1", "branch2", "branch2" or something like that). Create another
(maybe bare) repo elsewhere, add it as remote to the three other ones
and push their branches upwards. Now you have 4 repos, 3 for working
on the individual branches and another for collecting them all (hub model).
You could also choose to throw the first three away and only work in
the last one.

(Since projects are virtually identical, a fix in one branch 
usually needs to be propagated to other branches)

In your case, cherry-pick might be the right for you.
You could also do a little bit refactoring, making a 4th branch which
the other 3 are then rebased onto. Then you could do your fixes in that
branch and merged into or rebase the other 3 onto that one.

My second question is that each branch has a huge folder with image data. 
By huge I mean 1 to 4Gb, depending on the branch.  Since images are not 
directly relevant to the development work, is there a way to not include 
those folders in git?

see .gitignore file.
nevertheless it might be useful to also have all the images in the
repo for backup reasons.

BTW: if you're concerned about disk space, you could add the object dir
of the 4th (hub) repository to the 3 working repos (run git-gc in the
hub repo before that!). Next gc runs will remove the objects that are
already present in the hub. But beware! If you remove something in the
hub repo and run git-gc there, you could loose objects in the other repos!
(maybe it would be wise to add the 3 working repos as remotes in the
hub and always run an git remote update before git-gc in the hub).


cu
-- 
----------------------------------------------------------------------
 Enrico Weigelt, metux IT service -- http://www.metux.de/

 phone:  +49 36207 519931  email: weigelt@metux.de
 mobile: +49 151 27565287  icq:   210169427         skype: nekrad666
----------------------------------------------------------------------
 Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme
----------------------------------------------------------------------

(+cc: Dennis again, Alex)

Hi,

Enrico Weigelt wrote:

* Dennis [off-list ref] wrote:

quoted

Now, this can be either really simple or really complicated.  My first 
question is:  how do I set the repository up in the proper way where I 
could work on all 3 projects separately, with additional possibility of 
working on branch1 only and later committing my changes to branch2 and 
branch3.  

As first step you could create 3 separate git repos in each directory
and add everything to it (git init, git add -A, git commit). Then 
rename the branches properly (so instead of "master", they'll be called
"branch1", "branch2", "branch2" or something like that). Create another
(maybe bare) repo elsewhere, add it as remote to the three other ones
and push their branches upwards.

So this looks like so:

	for i in project1 project2 project3
	do
		(
			cd "$i"
			git init
			git add .
			git commit
		)
	done
	git init main
	cd main
	for i in project1 project2 project3
	do
		git fetch ../$i master:$i
	done
	mv project1 project2 project3 away/

If you would like multiple worktrees (one for each branch, maybe) for
the main repo, you might want to look into the new-workdir script in
contrib/workdir (but do consider the caveats[1]).

quoted

(Since projects are virtually identical, a fix in one branch 
usually needs to be propagated to other branches)

In your case, cherry-pick might be the right for you.

e.g., when project3 gets a new fix:

	git checkout project1
	git cherry-pick project3

You could also do a little bit refactoring, making a 4th branch which
the other 3 are then rebased onto.

Right, what is the actual relationship between these projects?  Do
they actually represent branches in the history of a single project?

Suppose project1 is historically an ancestor to project2, project3,
and project4, which are independent.  (Maybe project1 is the initial
version and projects 2,3,4 are ports to other platforms.)  You could
take this into account when initially setting up the branches, like
this:

	git init main
	cd main
	GIT_DIR=$(pwd)/.git; export GIT_DIR
	GIT_WORK_TREE=../project1 git add .
	GIT_WORK_TREE=../project1 git commit
	git branch -m project1
	for i in project2 project3 project4
	do
		git checkout -b $i project1
		GIT_WORK_TREE=../$i git add -A
		GIT_WORK_TREE=../$i git commit
	done

(and use gitk --all when done to make sure everything looks right)

Alternatively, you can rearrange the history afterwards:

	$ git cat-file commit project2 | tee project2
	tree 76db51024713f6ef191928a8445d48d39ab55434
	author Junio C Hamano [off-list ref] 1289324716 -0800
	committer Junio C Hamano [off-list ref] 1289324716 -0800

	project2: an excellent project
	$ git rev-parse project1
	$ vi project2
	... add a "parent <object id>" line
	    after the tree line,
	    where <object id> is the full object name rev-parse printed ...
	$ git hash-object -t commit -w project2
	$ git branch -f branch2 <the object name hash-object prints>
	... repeat for project3 and project4 ...
	$ gitk --all;		# to make sure everything looks right

This is less convenient than it ought to be.  It would be nice to add
a "git graft" command to automate this procedure, which

 - interacts well with "git replace"
 - doesn't interact poorly with "git fetch" like .git/info/grafts does
 - could be more convenient to use than .git/info/grafts.

As the gitworkflows man page mentions, if you make your fixes on the
oldest branch they apply to (project1) and then merge to all later
branches, then the fixes will propagate forward correctly.  See the
"Graduation" and "Merging upwards" sections of gitworkflows for details.

quoted

My second question is that each branch has a huge folder with image data. 
By huge I mean 1 to 4Gb, depending on the branch.  Since images are not 
directly relevant to the development work, is there a way to not include 
those folders in git?

I would suggest tracking a symlink to another repository (or to a
directory tracked through other means, like unison).

Hope that helps,
Jonathan

[1] If you have two worktrees for the same project with the
same branch checked out at a given moment, the results can be
confusing (changes made in one worktree will look like they have
been commited and undone in the other).

The "detached HEAD" feature (which git-checkout.1 explains) and
multiple worktrees do not interact so well: the need to preserve
commits while no branch was checked out in one worktree will not be
taken into account when "git gc" runs (explicitly or implicitly!) on
the other.  This can be very disconcerting.