This page is no longer maintained — Please continue to the home page at www.scala-lang.org

Using Hudson as our "build must always work" bot

13 replies
Joshua.Suereth
Joined: 2008-09-02,
User offline. Last seen 32 weeks 5 days ago.
I've been doing some research.  I don't think we can use Hudson's git plugin because it's designed for an alternative workflow than we're using[1].  E.g. the 2.9.x and 2.8.x branches would cause problems.

As such, I'd like to take a modified approach to what Adriaan has done for LMS.   Here's my two proposals:

PROPOSAL #1: RIGOR IS SLICKER
(1) All pull-requests on scala/scala go through a "develop" branch (similarly for 2.9.x and 2.8.x)(2) A "checkin" build on Jenkins pulls in the develop branch and runs the nightly build.  If it succeeds it pushes the commits to a "validation" branch (similarly for 2.9.x and 2.8.x).    NOTE: This branch can live in an alternative github location, like "scala/jenkins-bot-scala" (3-N) We repeat #2 but for the windows build, the JVM build etc.   Each time we get another build passing, we push the commits off.  (Ideally, there'd be a way to trigger a build only if *all* of these were successful). (N+1) Assuming all the platform tests are successful (JVMs, Windows, etc.), we push the "validation" branch into the public "master".  This is the only way things make it into the master branch.

PROPOSAL #2: BILL GATES WHO? (1) All pull-requests on scala/scala go through a "develop" branch (similarly for 2.9.x and 2.8.x)(2) A "checkin" build on Jenkins pulls in the develop branch and runs the nightly build.  If it succeeds it pushes the commits to a "master" branch (similarly for 2.9.x and 2.8.x). (3) Alternative platform builds occur, but we don't care as much as we would about a failing Scala build if we have a way to run the scala JARs and tests on those platforms.


I'm in favor of #1 since if we don't do it that way, the windows build will forever suffer.   If anyone has any alternative suggestions for how to do this, *please* let me know!
- Josh



[1] Here's an excerpt from the git plugin:

Advanced Features

Using Git, Jenkins and pre-build branch merging

Continuous Integration tools such as Jenkins are useful on projects as they give users early indication that a particular codebase is 'unstable' - and that if a developer checks it out, there will be trouble ahead (they won't be able to work on their own code, because someone else has broken something).

Unfortunately, by the time the build completes, this is often too late (particularly if the build cycle time is very long), as a developer has updated their working copy to the latest, unstable code in the repository and has begun work.

This can lead to the code base remaining unstable as developers tread on each others toes steadily fixing one thing, but breaking something else.

Some environments (e.g. TeamCity) attempt to fix this by making commits into SVN only 'really' happen once they have been tested. These kinds of 'delayed-commits' are problematic, because local SCM tools assume that commits will be immediately available, which can confuse them. In many ways this mechanism is a hack to get around the fact that branch management in SVN is very heavyweight.

Fortunately, with GIT and Jenkins, you can achieve the same 'stable branches' with minimal effort.

Set up your Jenkins project, and leave the 'branch' field in the Git SCM blank. This will cause Jenkins to consider any change on any branch for building.

Next, pick a particular branch name as the integration target in the 'Advanced' section - (e.g. 'master', or 'stable'), and select 'Merge before build'.

Select 'Push GIT tags back to origin repository' from the post-build actions (this is required to update your centralised git repo with the results of the build).

Now, developers should never commit directly to your integration branch (the 'master' or 'stable'). Instead, they should either use feature branches, or create new remote branches on commit (e.g : "git push origin HEAD:refs/heads/myNewFeature"). You could also set up your GIT repository to only accept commits onto the integration branch from Jenkins.

You're done. Commits should now be automatically merged with the integration branch (they will fail if they do not merge cleanly), and built. If the build succeeds, the result of the merge will be pushed back to the remote git repository.





Simon Ochsenreither
Joined: 2011-07-17,
User offline. Last seen 42 years 45 weeks ago.
Re: Using Hudson as our "build must always work" bot
I guess it doesn't really affect anyone outside those who can actually pull commits, but here are my 2 cent:

Getting patches pulled is already slow, so I'm not sure if another layer of complexity is worth it.

If the current work flow is not covered by the CI tool ... too bad, but if
“Unfortunately, by the time the build completes, this is often too late (particularly if the build cycle time is very long), as a developer has updated their working copy to the latest, unstable code in the repository and has begun work.”
is the problem we want to avoid, the solution is BUY FASTER HARDWARE, not try to satisfy some tools. Time of people is always more expensive than a one-time-investment in a fast machine.

I don't think it is a huge problem to run CI after the commit has hit trunk.

In fact, I never found that the issues mentioned are a huge problem. Everyone is expected to run the complete test suite before creating a pull request, so I'm not sure how this can avoid more problems at all.
extempore
Joined: 2008-12-17,
User offline. Last seen 35 weeks 3 days ago.
Re: Re: Using Hudson as our "build must always work" bot

On Thu, Dec 1, 2011 at 3:12 PM, Simon Ochsenreither
wrote:
> In fact, I never found that the issues mentioned are a huge problem.
> Everyone is expected to run the complete test suite before creating a pull
> request, so I'm not sure how this can avoid more problems at all.

Oh, everyone is expected to, then problem solved! Are you under the
impression that you've never sent a pull request which didn't pass the
test suite?

Geez, no wonder everybody thinks it's so easy. Try:

- about half of pull requests don't even compile
- at least half of the ones that do break a test
- that's before we even get to the subtle errors

And again, you are just as much a participant in those statistics as
everyone else, so I don't know what you're thinking here.

extempore
Joined: 2008-12-17,
User offline. Last seen 35 weeks 3 days ago.
Re: Using Hudson as our "build must always work" bot

On Thu, Dec 1, 2011 at 1:27 PM, Josh Suereth wrote:
> I'm in favor of #1 since if we don't do it that way, the windows build will
> forever suffer.   If anyone has any alternative suggestions for how to do
> this, *please* let me know!

We should do #1, but that shouldn't stop us from doing #2 first.

If the plugin world doesn't make this easy enough it's not much of a
shell script.

git fetch origin
git co -b test-run master
git merge develop && ant all.clean test # or whatever
if $?; then ; else ; fi

(Might have to fix partest exit codes etc. but point is, it's not a
high mountain.)

Joshua.Suereth
Joined: 2008-09-02,
User offline. Last seen 32 weeks 5 days ago.
Re: Re: Using Hudson as our "build must always work" bot
It's *exactly* right that time of people is more important than time of tools.  That's why submitting patches that break trunk is SO harmful.  There's too many committers for one person to be able to stop development from the rest.   If we don't have a working trunk, it hurts us in the long run.  
So if the dev workflow is "merge from master, issue pull requests to develop", then we're good to go.  We catch things before everyone has to deal with it, and the developers who break the build get to deal with it.
While automated tests won't catch all potential bugs in Scala, it will make the *cost* of accepting a patch much lower having them verified.  It should also improve our total throughput.
Also, commenting on pulling patches should only account for the past two days since the move to github.  I think the response time is pretty low right now.  We hope to keep it that way.  The automated tools are meant to aide in this speed.   The quality requirement for patches isn't changing, we're just trying to enforce as much as possible with automation so things progress faster.

- Josh

On Thu, Dec 1, 2011 at 6:12 PM, Simon Ochsenreither <simon [dot] ochsenreither [at] googlemail [dot] com> wrote:
I guess it doesn't really affect anyone outside those who can actually pull commits, but here are my 2 cent:

Getting patches pulled is already slow, so I'm not sure if another layer of complexity is worth it.

If the current work flow is not covered by the CI tool ... too bad, but if
“Unfortunately, by the time the build completes, this is often too late (particularly if the build cycle time is very long), as a developer has updated their working copy to the latest, unstable code in the repository and has begun work.”
is the problem we want to avoid, the solution is BUY FASTER HARDWARE, not try to satisfy some tools. Time of people is always more expensive than a one-time-investment in a fast machine.

I don't think it is a huge problem to run CI after the commit has hit trunk.

In fact, I never found that the issues mentioned are a huge problem. Everyone is expected to run the complete test suite before creating a pull request, so I'm not sure how this can avoid more problems at all.

Joshua.Suereth
Joined: 2008-09-02,
User offline. Last seen 32 weeks 5 days ago.
Re: Using Hudson as our "build must always work" bot
That's what I'm thinking for #2 for now.  Until we can get things building in other platforms.   Poor windows has the flakiest tests, so I almost want to point out how bad this is by failing to push changes when the build fails.   Combined with failing tests on OpenJDK, I think we'd shore up trunk in a hurry so that it remains relatively stable throughout development.

On Thu, Dec 1, 2011 at 6:28 PM, Paul Phillips <paulp [at] improving [dot] org> wrote:
On Thu, Dec 1, 2011 at 1:27 PM, Josh Suereth <joshua [dot] suereth [at] gmail [dot] com> wrote:
> I'm in favor of #1 since if we don't do it that way, the windows build will
> forever suffer.   If anyone has any alternative suggestions for how to do
> this, *please* let me know!

We should do #1, but that shouldn't stop us from doing #2 first.

If the plugin world doesn't make this easy enough it's not much of a
shell script.

git fetch origin
git co -b test-run master
git merge develop && ant all.clean test # or whatever
if $?; then <push to master>; else <send email yelling at someone>; fi

(Might have to fix partest exit codes etc. but point is, it's not a
high mountain.)

Blair Zajac
Joined: 2009-01-12,
User offline. Last seen 42 years 45 weeks ago.
Re: Using Hudson as our "build must always work" bot

On Dec 1, 2011, at 1:27 PM, Josh Suereth wrote:

> I've been doing some research. I don't think we can use Hudson's git plugin because it's designed for an alternative workflow than we're using[1]. E.g. the 2.9.x and 2.8.x branches would cause problems.
>
>
> As such, I'd like to take a modified approach to what Adriaan has done for LMS. Here's my two proposals:
>
>
> PROPOSAL #1: RIGOR IS SLICKER
>
> (1) All pull-requests on scala/scala go through a "develop" branch (similarly for 2.9.x and 2.8.x)
> (2) A "checkin" build on Jenkins pulls in the develop branch and runs the nightly build. If it succeeds it pushes the commits to a "validation" branch (similarly for 2.9.x and 2.8.x). NOTE: This branch can live in an alternative github location, like "scala/jenkins-bot-scala"
> (3-N) We repeat #2 but for the windows build, the JVM build etc. Each time we get another build passing, we push the commits off. (Ideally, there'd be a way to trigger a build only if *all* of these were successful).
> (N+1) Assuming all the platform tests are successful (JVMs, Windows, etc.), we push the "validation" branch into the public "master". This is the only way things make it into the master branch.
>
>
> PROPOSAL #2: BILL GATES WHO?
> (1) All pull-requests on scala/scala go through a "develop" branch (similarly for 2.9.x and 2.8.x)
> (2) A "checkin" build on Jenkins pulls in the develop branch and runs the nightly build. If it succeeds it pushes the commits to a "master" branch (similarly for 2.9.x and 2.8.x).
> (3) Alternative platform builds occur, but we don't care as much as we would about a failing Scala build if we have a way to run the scala JARs and tests on those platforms.

So both #1 and #2 accept the pull request and then CI is run? If that's the case, how are you going to manage multiple pull requests from different developers, merge all of them? What happens if there's a bad commit the middle?

Another completely different alternative is to use gerrit [1] as the repository for developers. Gerrit comes from Android's development team for a git based review system. After commits are reviewed on gerrit and committed, they can be pushed into github. Then you hook up Jenkins/Hudson to gerrit [2].

Now the cool thing about Gerrit/Jenkins integration is that commits pushed into gerrit can be checked by Jenkins/Hudson and verified even before they are reviewed or merged into master (or whichever branch the work is for). After the CI system verifies the build, a Scala committer could review them and then merge them into master or the branch the work is based off. Gerrit shows two columns for each commit, a review and a verified status. The verified status is updated by Jenkins and the review is done by a person. Also, any re-push of a change following a committers comments are rechecked.

We use this at work and it works pretty well.

Another advantage of this is that any outstanding changes in gerrit that haven't been merged can be rebased on the commits that have been merged and then retested with Jenkins to ensure there aren't any odd interactions.

Blair

[1] http://code.google.com/p/gerrit/
[2] https://wiki.jenkins-ci.org/display/JENKINS/Gerrit+Trigger

rytz
Joined: 2008-07-01,
User offline. Last seen 45 weeks 5 days ago.
Re: Using Hudson as our "build must always work" bot
One thing that worries me a bit is that not only the windows tests are flaky. For some timenow, we have two tests failing nondeterministically (timeofday, akka). Of course, theseshould be fixed. But we see that such situations happen, we cannot assure they won't happen in the future, and if they do, the commit chain should not be locked.

On Fri, Dec 2, 2011 at 02:10, Josh Suereth <joshua [dot] suereth [at] gmail [dot] com> wrote:
That's what I'm thinking for #2 for now.  Until we can get things building in other platforms.   Poor windows has the flakiest tests, so I almost want to point out how bad this is by failing to push changes when the build fails.   Combined with failing tests on OpenJDK, I think we'd shore up trunk in a hurry so that it remains relatively stable throughout development.

On Thu, Dec 1, 2011 at 6:28 PM, Paul Phillips <paulp [at] improving [dot] org> wrote:
On Thu, Dec 1, 2011 at 1:27 PM, Josh Suereth <joshua [dot] suereth [at] gmail [dot] com> wrote:
> I'm in favor of #1 since if we don't do it that way, the windows build will
> forever suffer.   If anyone has any alternative suggestions for how to do
> this, *please* let me know!

We should do #1, but that shouldn't stop us from doing #2 first.

If the plugin world doesn't make this easy enough it's not much of a
shell script.

git fetch origin
git co -b test-run master
git merge develop && ant all.clean test # or whatever
if $?; then <push to master>; else <send email yelling at someone>; fi

(Might have to fix partest exit codes etc. but point is, it's not a
high mountain.)


Stefan Zeiger
Joined: 2008-12-21,
User offline. Last seen 27 weeks 3 days ago.
Re: Using Hudson as our "build must always work" bot
On 2011-12-01 22:27, Josh Suereth wrote:
CAFLqJkxZ2M0D0MC_5BYBJdnqanVjyCFVu0gcew0NO6UifAp+zQ [at] mail [dot] gmail [dot] com" type="cite">PROPOSAL #1: RIGOR IS SLICKER
(1) All pull-requests on scala/scala go through a "develop" branch (similarly for 2.9.x and 2.8.x) (2) A "checkin" build on Jenkins pulls in the develop branch and runs the nightly build.  If it succeeds it pushes the commits to a "validation" branch (similarly for 2.9.x and 2.8.x).    NOTE: This branch can live in an alternative github location, like "scala/jenkins-bot-scala" (3-N) We repeat #2 but for the windows build, the JVM build etc.   Each time we get another build passing, we push the commits off.  (Ideally, there'd be a way to trigger a build only if *all* of these were successful). (N+1) Assuming all the platform tests are successful (JVMs, Windows, etc.), we push the "validation" branch into the public "master".  This is the only way things make it into the master branch.

Would it be possible do that without all the validation branches, just running the different builds in parallel and pushing to master (or handing them off for a manual review) if all builds succeed?

-sz
adriaanm
Joined: 2010-02-08,
User offline. Last seen 31 weeks 4 days ago.
Re: Using Hudson as our "build must always work" bot


On Fri, Dec 2, 2011 at 8:55 AM, Lukas Rytz <lukas [dot] rytz [at] epfl [dot] ch> wrote:
the commit chain should not be locked.
I agree timely builds for each commit and a stable test suite are important,but the commit chain has a bit of slack now :-)
We should sets things up so that you can pull from the integration branch if you want to work on untested commits.
If all tests pass, you'll already be up to date with master. At least, assuming we'll merge these commits into master rather than rebasing, but I think merging's what we should do. There's the thorny issue of fast-forward merging, and undoing merges of pull requests that are rejected, but i think having a common branch for all cutting-edge features is beneficial to test for feature interaction with the features you are working on.
So, when a pull request is rejected, the merge will be undone and you'll need to rebase any branches that merged the integration branch after the rejected requested was merged (since those commits from the integration branch will not end up in master).
That's the price you pay -- I don't think we can avoid that.
Grzegorz pointed out during lunch that we don't really need an integration branch -- we could all just pull all the pull requests pro-actively. Also, we need to plan for master diverging during review/testing, which may cause accepted pull requests to still fail to merge cleanly. When that happens, who cleans it up? If it's the reviewer, it's motivation for quick (though dirty?) reviews, but contributors may be put off by problems like this, since the conflict is most likely not their fault.


cheersadriaan
adriaanm
Joined: 2010-02-08,
User offline. Last seen 31 weeks 4 days ago.
Re: Using Hudson as our "build must always work" bot
Would it be possible do that without all the validation branches, just running the different builds in parallel and pushing to master (or handing them off for a manual review) if all builds succeed?
yes, but that would reduce the incentive for developers to be on develop (since the equivalent is tedious: pull all pending pull requests yourself) rather than master, which would delay detecting feature interaction
also, there's another issue we haven't really discussed: in principle, only commits destined for the next release should go into master
this stable/staging hierarchy could be repeated for long-running projects, such as scala-virtualized, scala-macros, ...
cheersadriaan
Stefan Zeiger
Joined: 2008-12-21,
User offline. Last seen 27 weeks 3 days ago.
Re: Using Hudson as our "build must always work" bot
On 2011-12-02 14:18, Adriaan Moors wrote:
H96Djig-DuKtXEEg4B4AA [at] mail [dot] gmail [dot] com" type="cite">
Would it be possible do that without all the validation branches, just running the different builds in parallel and pushing to master (or handing them off for a manual review) if all builds succeed?
yes, but that would reduce the incentive for developers to be on develop (since the equivalent is tedious: pull all pending pull requests yourself) rather than master, which would delay detecting feature interaction

Oh, right, since I just ran into the issue of what to base a follow-up-branch on, I now understand the purpose of these branches. What I was actually referring to is the sequential pipeline for the different builds. Would anyone want to use these branches? Otherwise I'd prefer to run all the integration tests in parallel.

H96Djig-DuKtXEEg4B4AA [at] mail [dot] gmail [dot] com" type="cite"> this stable/staging hierarchy could be repeated for long-running projects, such as scala-virtualized, scala-macros, ...

What's happening to those projects anyway? Were they already on git? Has their history been rebased to the new scala/scala repository?

-sz
adriaanm
Joined: 2010-02-08,
User offline. Last seen 31 weeks 4 days ago.
Re: Using Hudson as our "build must always work" bot
Oh, right, since I just ran into the issue of what to base a follow-up-branch on, I now understand the purpose of these branches. What I was actually referring to is the sequential pipeline for the different builds. Would anyone want to use these branches? Otherwise I'd prefer to run all the integration tests in parallel.
sure, good point -- we should reduce latency in any way we canjenkins supports parallel jobs, so that's an orthogonal issue, as I understand it
 
this stable/staging hierarchy could be repeated for long-running projects, such as scala-virtualized, scala-macros, ...

What's happening to those projects anyway? Were they already on git? Has their history been rebased to the new scala/scala repository?
scala-virtualized will be rebased into a couple of chunks, so most of its history will be lost, but I don't think that's a bad thing scala-macros is relatively recent, so I guess this'll be less of a problem there
adriaan
Joshua.Suereth
Joined: 2008-09-02,
User offline. Last seen 32 weeks 5 days ago.
Re: Using Hudson as our "build must always work" bot
We can also have long-running projects like scala-virtualized and others have their own scala/* repository that we can set up a similar workflow on.  This gives thoses projects a clear "head" for people to integrate into and they should cleanly merge with master...
They also have their own Pull-requests, so owners of those features can focus on things before the feature makes a pull-request on main-scala.  We can also cherry-pick commits into branches for pull requests to main scala as needed...

I think we need to take flaky tests very seriously here. If a test is flaky, we need to unflake it quickly.   Trunk being in an unbuildable state slows down too many people for us to allow it.  I think having a 'unstable' develop branch and a 'stable' master here are *good things* (tm).  It means that the nightlies/milestones that the community can test will be far more stable than ever before.   We might be able to get more in-process feedback on features or performances regressions (before we get the performance test suite up).
- Josh

On Fri, Dec 2, 2011 at 8:43 AM, Adriaan Moors <adriaan [dot] moors [at] epfl [dot] ch> wrote:
Oh, right, since I just ran into the issue of what to base a follow-up-branch on, I now understand the purpose of these branches. What I was actually referring to is the sequential pipeline for the different builds. Would anyone want to use these branches? Otherwise I'd prefer to run all the integration tests in parallel.
sure, good point -- we should reduce latency in any way we canjenkins supports parallel jobs, so that's an orthogonal issue, as I understand it
 
this stable/staging hierarchy could be repeated for long-running projects, such as scala-virtualized, scala-macros, ...

What's happening to those projects anyway? Were they already on git? Has their history been rebased to the new scala/scala repository?
scala-virtualized will be rebased into a couple of chunks, so most of its history will be lost, but I don't think that's a bad thing scala-macros is relatively recent, so I guess this'll be less of a problem there
adriaan

Copyright © 2012 École Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland