Coping with the pull-before-you-push model

  After a deployment of Git on a centralized server at my place of 
business, the largest amount of grumbling has been with the 
pull-before-you-push model.  Coming from the file-centric Perforce where 
you need only have latest of just the files you are submitting, the 
pull-before-you-push model has really been a pain in the neck for a 
large team.

Even with topic branches being used, merges to master occur frequently.  
It can really be a frustrating battle to get your merged branch pushed 
to the central master branch.  In the time it took you to pull, test, 
and push, someone has probably already pushed before you.  To cope with 
this, people will pull, not bother testing, and immediately push their 
changes.  Yes, this could result in build instability, but it is 
considered better than never being able to make your change live.

(Let's ignore what we should or shouldn't be doing as far as 
'development practices'.  :)  We're solving the problems one step at a 
time...)

Gerrit provides a compelling model where branches are pushed to the code 
review server in the form refs/for/master, and the given push will 
always succeed.  Code reviews are performed, someone sets the verified 
bit, and the change is submitted and merged to master by Gerrit itself 
in a queued fashion.  Unfortunately, its general "requirement" to squash 
your branch down to a single commit is, possibly, a showstopper.  If it 
treated a branch merge as a group of commits that MUST stay together, 
that would be perfect.

What other tools are out there that would let users successfully push 
their branch to the server (without having the HEAD master commit), and 
the push would be automatically merged to the master branch?

Is there another workflow that is successful for your large(-ish) 
enterprise team?

Thanks for your insights!

Josh

On Thu, Sep 9, 2010 at 04:47, Joshua Jensen [off-list ref] wrote:

 After a deployment of Git on a centralized server at my place of business,
the largest amount of grumbling has been with the pull-before-you-push
model.  Coming from the file-centric Perforce where you need only have
latest of just the files you are submitting, the pull-before-you-push model
has really been a pain in the neck for a large team.

Even with topic branches being used, merges to master occur frequently.  It
can really be a frustrating battle to get your merged branch pushed to the
central master branch.  In the time it took you to pull, test, and push,
someone has probably already pushed before you.  To cope with this, people
will pull, not bother testing, and immediately push their changes.  Yes,
this could result in build instability, but it is considered better than
never being able to make your change live.

(Let's ignore what we should or shouldn't be doing as far as 'development
practices'.  :)  We're solving the problems one step at a time...)

Let's not ignore that.

Presumably you had exactly the same problem in perforce, i.e. because
you only had have the files you were changing checked out in Perforce
in the time between `hack && pull && test && push` someone else might
have already pushed. Thus what you just submitted wasn't guaranteed to
pass tests.

So is the flow in Git where you don't run the tests again, rebase and
push and hope for the best any different?

Gerrit provides a compelling model where branches are pushed to the code
review server in the form refs/for/master, and the given push will always
succeed.  Code reviews are performed, someone sets the verified bit, and the
change is submitted and merged to master by Gerrit itself in a queued
fashion.  Unfortunately, its general "requirement" to squash your branch
down to a single commit is, possibly, a showstopper.  If it treated a branch
merge as a group of commits that MUST stay together, that would be perfect.

This sounds like something that's configurable in Gerrit, or should
be.

[..]

Is there another workflow that is successful for your large(-ish) enterprise
team?

Linux manages to deal with a huge number of commits, but does so by
having subsystems.

Maybe that's something you can use in your codebase?

  ----- Original Message -----
From: Ævar Arnfjörð Bjarmason
Date: 9/9/2010 7:06 AM

On Thu, Sep 9, 2010 at 04:47, Joshua Jensen[off-list ref]  wrote:

quoted

  After a deployment of Git on a centralized server at my place of business,
the largest amount of grumbling has been with the pull-before-you-push
model.  Coming from the file-centric Perforce where you need only have
latest of just the files you are submitting, the pull-before-you-push model
has really been a pain in the neck for a large team.

Even with topic branches being used, merges to master occur frequently.  It
can really be a frustrating battle to get your merged branch pushed to the
central master branch.  In the time it took you to pull, test, and push,
someone has probably already pushed before you.  To cope with this, people
will pull, not bother testing, and immediately push their changes.  Yes,
this could result in build instability, but it is considered better than
never being able to make your change live.

(Let's ignore what we should or shouldn't be doing as far as 'development
practices'.  :)  We're solving the problems one step at a time...)

Let's not ignore that.

Fair.

Presumably you had exactly the same problem in perforce, i.e. because
you only had have the files you were changing checked out in Perforce
in the time between `hack&&  pull&&  test&&  push` someone else might
have already pushed. Thus what you just submitted wasn't guaranteed to
pass tests.

So is the flow in Git where you don't run the tests again, rebase and
push and hope for the best any different?

The end result is the same; submitted code is never really tested 
against latest in Perforce either.  The primary difference between the 
two is that the Perforce submit is successful the majority of the time 
(odds of someone editing and submitting the same file as you are low), 
and the Git push fails the majority of the time.

Don't get me wrong.  I've given training on why Git's enforced 
pull-before-you-push model can be better than what we had before 
(reproducible state, fewer broken builds, etc).  Nevertheless, the issue 
is very frequent, and that's why I am querying others.

quoted

Gerrit provides a compelling model where branches are pushed to the code
review server in the form refs/for/master, and the given push will always
succeed.  Code reviews are performed, someone sets the verified bit, and the
change is submitted and merged to master by Gerrit itself in a queued
fashion.  Unfortunately, its general "requirement" to squash your branch
down to a single commit is, possibly, a showstopper.  If it treated a branch
merge as a group of commits that MUST stay together, that would be perfect.

This sounds like something that's configurable in Gerrit, or should
be.

Agreed, but it appears it is currently a missing feature.  There have 
been discussions about it on their mailing list over the past months, 
and a feature request is in their tracker.  I am unsure of their 
progress or even interest.

quoted

[..]

Is there another workflow that is successful for your large(-ish) enterprise
team?

Linux manages to deal with a huge number of commits, but does so by
having subsystems.

Maybe that's something you can use in your codebase?

Unless I'm mistaken, though, it seems to work in reverse of what our 
working model has been for years.

I'm grossly oversimplifying the process, but the Linux model seems to be 
built on hierarchical 'pull requests'.  I can tell a subsystem 
maintainer I have some changes and then ask that maintainer to pull them 
from a certain location.  When that person has time/inclination, the 
change is pulled, merged in, and then another pull request is sent to 
the upstream hierarchy.

This _is_ compelling, but even if it would work within the company I 
work for, it is such a dramatic shift in workflow that I am certain it 
could not be done in one fell swoop.

Thanks for your insights!

Josh

On Fri, Sep 10, 2010 at 12:43 AM, Joshua Jensen
[off-list ref] wrote:>>

quoted

Presumably you had exactly the same problem in perforce, i.e. because
you only had have the files you were changing checked out in Perforce
in the time between `hack&&  pull&&  test&&  push` someone else might
have already pushed. Thus what you just submitted wasn't guaranteed to
pass tests.

So is the flow in Git where you don't run the tests again, rebase and
push and hope for the best any different?

The end result is the same; submitted code is never really tested against
latest in Perforce either.  The primary difference between the two is that
the Perforce submit is successful the majority of the time (odds of someone
editing and submitting the same file as you are low), and the Git push fails
the majority of the time.

Don't get me wrong.  I've given training on why Git's enforced
pull-before-you-push model can be better than what we had before
(reproducible state, fewer broken builds, etc).  Nevertheless, the issue is
very frequent, and that's why I am querying others.

I am not sure that git does enforce a pull before push model, except
if you attempt
to use git without the equivalent of a maintainer - in essence
devolving responsibility of maintaining the tip of the shared branch
to the whole team.

This is probably the cultural shift that is hardest for enterprises to
accept - why do I need a _person_ to do this _manual_ work when tools
like {insert favourite non-DVCS here} can do this for me? To
management, this looks like a step-backwards.

In our case, the team leaders act like the subsystem leads and the
build team acts like the maintainer. Team leads have a more review
oriented focus, where as the build team doesn't understand the code
that well and will reject merge conflicts if they are non-trivial.
Their focus is just to keep the daily build rhythm going.

If you don't have a maintainer role, then everyone is obliged to pull,
test, pull-again, then push because there is no-one else there to do
it and no other time to do it. The delays inherent in the test process
inevitably mean you have to cycle several times until your commit
wins.

But with a maintainer role on your project, you don't need to do this.
Everyone develops on yesterday's baseline and verifies that they don't
regress anything with respect to that baseline - there is no
requirement to certify against the bleeding edge of the integration
stream - that's the role of the maintainer/build team once the daily
build is done, thereby amortising test execution effort across the
whole team.

I'm grossly oversimplifying the process, but the Linux model seems to be
built on hierarchical 'pull requests'.  I can tell a subsystem maintainer I
have some changes and then ask that maintainer to pull them from a certain
location.  When that person has time/inclination, the change is pulled,
merged in, and then another pull request is sent to the upstream hierarchy.

Note that one difference between the maintainer and maintainer-less
model is that the tip of the reference branch is published relatively
rarely - it is quite stable. This is quite helpful because the
baseline doesn't keep shifting every 30 minutes. In the
maintainer-less model, you never have stability, nor any opportunity
to acquire it.

This _is_ compelling, but even if it would work within the company I work
for, it is such a dramatic shift in workflow that I am certain it could not
be done in one fell swoo

The big difference between commercial work and open source projects
like git and linux is that the latter have just one constraint -
quality whereas the former also have budgets and schedules to worry
about. Unintegrated crap code has cost the open source project almost
nothing. Commercial enterprises, on the other hand, pay lots of money
for people to develop code whether it is crap or otherwise. The idea
of leaving expensive code unintegrated causes management paroxysms of
concern that are hard to ignore - a tool that blindly integrates crap
code automatically is a soothing balm for such people. Hence the
resistance to tools like git that encourage a maintainer role and,
implicit in that, the possibility of review that might result in crap
code being exposed for what it is.

jon seymour.

On Fri, Sep 10, 2010 at 03:35:21PM +1000, Jon Seymour wrote:

This is probably the cultural shift that is hardest for enterprises to
accept - why do I need a _person_ to do this _manual_ work when tools
like {insert favourite non-DVCS here} can do this for me? To
management, this looks like a step-backwards.

Bear in mind that you can still shift to a maintainer model, but keep
the maintainer automated. That is, you can queue up "to-pull" heads, and
then have an automated process pull them one by one and do some basic QA
(does it merge, does it build, does it pass automated tests, etc). Which
is not that different from what many shops do in the non-maintainer
model, except that when you break the build, the maintainer process
notices _before_ publishing the merged tip, so everybody won't try to
build on your broken crap.

I seem to recall that Gerrit does something like this, but I may be
mis-remembering. I haven't actually used it for real work.

I still prefer a human maintainer, because they can do things like
reorder the queue manually (or outright reject flaky topics) to get more
sensible merges, or do easy but non-trivial merges themselves to avoid
kicking code back to the developer.

-Peff

  ----- Original Message -----
From: Jon Seymour
Date: 9/9/2010 11:35 PM

quoted

This _is_ compelling, but even if it would work within the company I work
for, it is such a dramatic shift in workflow that I am certain it could not
be done in one fell swo

The big difference between commercial work and open source projects
like git and linux is that the latter have just one constraint -
quality whereas the former also have budgets and schedules to worry
about. Unintegrated crap code has cost the open source project almost
nothing. Commercial enterprises, on the other hand, pay lots of money
for people to develop code whether it is crap or otherwise. The idea
of leaving expensive code unintegrated causes management paroxysms of
concern that are hard to ignore - a tool that blindly integrates crap
code automatically is a soothing balm for such people. Hence the
resistance to tools like git that encourage a maintainer role and,
implicit in that, the possibility of review that might result in crap
code being exposed for what it is.

My apologies for the late response.  I'm not ignoring this.  I am trying 
to process it in the context of my organization as I establish the 
workflows we'll be using.  I may have some follow up questions soon.

Josh

  ----- Original Message -----
From: Jeff King
Date: 9/10/2010 8:15 AM

On Fri, Sep 10, 2010 at 03:35:21PM +1000, Jon Seymour wrote:

quoted

This is probably the cultural shift that is hardest for enterprises to
accept - why do I need a _person_ to do this _manual_ work when tools
like {insert favourite non-DVCS here} can do this for me? To
management, this looks like a step-backwards.

Bear in mind that you can still shift to a maintainer model, but keep
the maintainer automated. That is, you can queue up "to-pull" heads, and
then have an automated process pull them one by one and do some basic QA
(does it merge, does it build, does it pass automated tests, etc). Which
is not that different from what many shops do in the non-maintainer
model, except that when you break the build, the maintainer process
notices _before_ publishing the merged tip, so everybody won't try to
build on your broken crap.

Do you know of any existing software that does this?  This may be ideal 
in the short term.

I still prefer a human maintainer, because they can do things like
reorder the queue manually (or outright reject flaky topics) to get more
sensible merges, or do easy but non-trivial merges themselves to avoid
kicking code back to the developer.

I agree with you concerning how valuable a human maintainer is.

Josh

On Mon, Sep 13, 2010 at 10:47:30PM -0600, Joshua Jensen wrote:

quoted

Bear in mind that you can still shift to a maintainer model, but keep
the maintainer automated. That is, you can queue up "to-pull" heads, and
then have an automated process pull them one by one and do some basic QA
(does it merge, does it build, does it pass automated tests, etc). Which
is not that different from what many shops do in the non-maintainer
model, except that when you break the build, the maintainer process
notices _before_ publishing the merged tip, so everybody won't try to
build on your broken crap.

Do you know of any existing software that does this?  This may be
ideal in the short term.

I think that Avery Pennarun's gitbuilder may do what you want:

  http://github.com/apenwarr/gitbuilder/

but I've never used it.

I seem to recall from one of Shawn's presentations on Gerrit Code Review
that it does something like this, too, but I can't seem to find any docs
about it in my brief search:

  http://code.google.com/p/gerrit/

It may be that Gerrit doesn't handle building itself, but that the
Android project is running something alongside it. Shawn may be able to
say more.

Basically, what we are talking about is continuous integration, with the
slight twist that instead of developers pushing commits to a mainline
branch which is built and tested, we would build and test their commits
and then merge them to the mainline branch.

Systems like Hudson that do continuous integration and support git may
handle a workflow like this, but I don't know (I've only ever used
Hudson in the everything-goes-to-svn-trunk model).

There is also a small continuous integration system that lives in the
contrib/continuous directory of git.git itself. It's quite old at this
point, but I wouldn't be surprised if its descendant is what Gerrit
uses.

-Peff

On Mon, Sep 13, 2010 at 10:24 PM, Jeff King [off-list ref] wrote:

On Mon, Sep 13, 2010 at 10:47:30PM -0600, Joshua Jensen wrote:

quoted

quoted

Bear in mind that you can still shift to a maintainer model, but keep
the maintainer automated. That is, you can queue up "to-pull" heads, and
then have an automated process pull them one by one and do some basic QA
(does it merge, does it build, does it pass automated tests, etc). Which
is not that different from what many shops do in the non-maintainer
model, except that when you break the build, the maintainer process
notices _before_ publishing the merged tip, so everybody won't try to
build on your broken crap.

Do you know of any existing software that does this?  This may be
ideal in the short term.

I think that Avery Pennarun's gitbuilder may do what you want:

 http://github.com/apenwarr/gitbuilder/

but I've never used it.

gitbuilder doesn't do this.

However, it does let you automatically do user-defined things for each
branch, and it stores the result for each one.  So it would be a
pretty short trip from there to auto-merging when a branch is ready.

Have fun,

Avery

On Sep 14, 2010, at 12:47 AM, Joshua Jensen wrote:

quoted

quoted

management, this looks like a step-backwards.

Bear in mind that you can still shift to a maintainer model, but keep
the maintainer automated. That is, you can queue up "to-pull" heads, and
then have an automated process pull them one by one and do some basic QA
(does it merge, does it build, does it pass automated tests, etc). Which
is not that different from what many shops do in the non-maintainer
model, except that when you break the build, the maintainer process
notices _before_ publishing the merged tip, so everybody won't try to
build on your broken crap.

Do you know of any existing software that does this?  This may be ideal in the short term.

Our workflow at $WORK involves pushing changes to gerrit to various "effort branches", and then once they are approved, we have a "Mergitator" script that will attempt to merge the effort branch with the merged master branch, and then attempt to do a build.  If the build succeeds, then the changes will get pushed back to the publically visible merged master branch, and then the Mergitator will move on to the next effort branch that requires merging.   If there is a merge conflict, the Mergitator will refuse the merge, and then give instructions on how to fix up the tree to avoid merge conflicts.

The Mergitator code hasn't been released, and I suspect the main reason is that there's relatively little code that could be used outside of our environment, and a large amount of code which contains lots of details about our internal build system that would have to be stripped out and generalized before it could be released --- and no one has time to do it.

So this probably doesn't help you since I suspect you meant to ask the question, "do you know of any existing publically available software", but I can tell you that it certainly can be done, and that software exists.  Making it be software which is useful and usable to you would definitely take more work...

-- Ted

  ----- Original Message -----
From: Theodore Tso
Date: 9/14/2010 6:12 AM

On Sep 14, 2010, at 12:47 AM, Joshua Jensen wrote:

quoted

quoted

Bear in mind that you can still shift to a maintainer model, but keep
the maintainer automated. That is, you can queue up "to-pull" heads, and
then have an automated process pull them one by one and do some basic QA
(does it merge, does it build, does it pass automated tests, etc). Which

Do you know of any existing software that does this?  This may be ideal in the short ter

Our workflow at $WORK involves pushing changes to gerrit to various "effort branches", and then once they are approved, we have a "Mergitator" script that will attempt to merge the effort branch with the merged master branch, and then attempt to do a build.  If the build succeeds, then the changes will get pushed back to the publically visible merged master branch, and then the Mergitator will move on to the next effort branch that requires merging.   If there is a merge conflict, the Mergitator will refuse the merge, and then give instructions on how to fix up the tree to avoid merge conflicts.

How does the integration with Gerrit work here?  The only thing that 
comes to mind is that you do something like:

git push gerrit HEAD:refs/for/merged-master

Then the approvals are done.  Afterward, Gerrit merges to the 
merged-master branch.

I would suppose an external script is performing regular fetches.  When 
it sees new code, it builds.

No, this can't be right, but I'll leave my incorrect workflow here.

So this probably doesn't help you since I suspect you meant to ask the question, "do you know of any existing publically available software", but I can tell you that it certainly can be done, and that software exists.  Making it be software which is useful and usable to you would definitely take more work...

It's the branch queueing issue that is my current hang up.  Gerrit's 
method is slick, but that won't work outside of JGit.  I'm not opposed 
to JGit; I just haven't touched Java in years.

So, perhaps, a web interface where the branch owner selects the (pushed 
to central server) branch name that is ready to go.  A merge is 
attempted.  If it succeeds, great.  If it fails, then the merge is reset?

Josh

On Tue, Sep 14, 2010 at 11:51 AM, Joshua Jensen
[off-list ref] wrote:

 ----- Original Message -----
From: Theodore Tso
Date: 9/14/2010 6:12 AM

quoted

On Sep 14, 2010, at 12:47 AM, Joshua Jensen wrote:

quoted

quoted

Bear in mind that you can still shift to a maintainer model, but keep
the maintainer automated. That is, you can queue up "to-pull" heads, and
then have an automated process pull them one by one and do some basic QA
(does it merge, does it build, does it pass automated tests, etc). Which

Do you know of any existing software that does this?  This may be ideal
in the short ter

Our workflow at $WORK involves pushing changes to gerrit to various
"effort branches", and then once they are approved, we have a "Mergitator"
script that will attempt to merge the effort branch with the merged master
branch, and then attempt to do a build.  If the build succeeds, then the
changes will get pushed back to the publically visible merged master branch,
and then the Mergitator will move on to the next effort branch that requires
merging.   If there is a merge conflict, the Mergitator will refuse the
merge, and then give instructions on how to fix up the tree to avoid merge
conflicts.

How does the integration with Gerrit work here?  The only thing that comes
to mind is that you do something like:

git push gerrit HEAD:refs/for/merged-master

Then the approvals are done.  Afterward, Gerrit merges to the merged-master
branch.

I would suppose an external script is performing regular fetches.  When it
sees new code, it builds.

No, this can't be right, but I'll leave my incorrect workflow here.

quoted

So this probably doesn't help you since I suspect you meant to ask the
question, "do you know of any existing publically available software", but I
can tell you that it certainly can be done, and that software exists.
 Making it be software which is useful and usable to you would definitely
take more work...

It's the branch queueing issue that is my current hang up.  Gerrit's method
is slick, but that won't work outside of JGit.  I'm not opposed to JGit; I
just haven't touched Java in years.

So, perhaps, a web interface where the branch owner selects the (pushed to
central server) branch name that is ready to go.  A merge is attempted.  If
it succeeds, great.  If it fails, then the merge is reset?

Josh
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

In our case we have moved from CVS to git and had pretty much same
problems.  The only difference was that we are working with java and
trying to have every project in separate repository. So we don't have
many commits into common codebase. We have a server keeping bare
mainline repos.
In order to integrate the project with others we are using Hudson CI
which performs builds upon each change detected on the server side
(i.e. mainline repository) and also builds the projects that are
dependent on the one you just changed. Those dependencies can be
described inside each hudson job, but we are using Ivy for dependency
management. If you have good testing infrastructure that is even
better.

So generally the amount of contributors to one project is pretty
small. Some projects are solo, some have small teams coworking on one
project code base.

We are using two main branch model.
The development goes in topic branches. The qa branch is a staging
branch for the changes to be accumulated for the release. People are
using direct pushes to qa branch to integrate or a maintainer. Hudson
builds this codebase. When all release functionality is done the
person responsible for the release is merging qa branch into the
master, which gets released (this must be a fast-forward).

So, i would say component oriented approach is the key here.

HTH,
Eugene

On Tue, Sep 14, 2010 at 09:51:22AM -0600, Joshua Jensen wrote:

How does the integration with Gerrit work here?  The only thing that
comes to mind is that you do something like:

git push gerrit HEAD:refs/for/merged-master

So we'll push to something to, say, refs/heads/fs/ext4/for-merged on
the gerrit server, and let gerrit do its thing.  After a colleague
approves all of the patches in the branch, gerrit will release the
commits to the branch, and only then will the mergitator script
attempt to do a trial merge of the effort branch with the
merged/master branch.  If the trial merge succeeds, then it will
attempt to do a trial compile.  If the trial compile succeeds then the
merged/master branch will be updated with the commit id of the trial
merge, and then the mergitator script will move on to the next effort
branch will has been updated.  If the mergitator fails to merge a
particular branch, then an e-mail is sent out explanining the cause of
the merge failure, so a human can fix things up.  This could be done
by cherry-picking a commit from merged/master which caused the merge
conflict, and then fixing up the merge conflict, for example.

A wise developer will do a trial merge on their own *before*
submitting their effort branch to gerrit for code review, but this is
not strictly speaking required.

						- Ted

On Tue, Sep 14, 2010 at 01:24:51AM -0400, Jeff King wrote:

I seem to recall from one of Shawn's presentations on Gerrit Code Review
that it does something like this, too, but I can't seem to find any docs
about it in my brief search:

  http://code.google.com/p/gerrit/

It may be that Gerrit doesn't handle building itself, but that the
Android project is running something alongside it. Shawn may be able to
say more.

Basically, what we are talking about is continuous integration, with the
slight twist that instead of developers pushing commits to a mainline
branch which is built and tested, we would build and test their commits
and then merge them to the mainline branch.

Gerrit doesn't do the build and test, but it isn't all that difficult
to hook into.  It also allows the results of the build and test to
record their state so that the developer can track what is happening.

The nice part about it, compared to other CI systems I've seen is that
it catches the problems before they are merged into master, rather
than after.

Internally, we actually do multiple levels of a CI-type thing.  Every
time a developer uploads a change, one machine performs a sanity build
on it (with multiple configurations).  These results are visible in
Gerrit, and it keeps people from putting too much effort into
reviewing code that doesn't even compile.

Once the code has gotten through two levels of code review, a more
throughough build and test system pulls the whole project (from
numerous git repos) builds and runs tests.  This takes long enough
that it doesn't do individual changes, so failures take work to track
down, but it does generally assure that the result of each 'git merge'
somewhat works.

The only real annoying part I've found with the gerrit model is that
the tree is filled with lots of merge comments (generally a merge for
every real commit).  The other option is to let Gerrit rebase, which
then gets annoying when a developer has pulled changes from other
developers.

It also has a nice link to a 'git pull' command to pull down an
individual change.

David