buildbot vs hudson/jenkins for C++ continuous integration
I’m currently using jenkins/hudson for continuous integration a large mostly C++ project. We have separate projects for trunk and every branch. Also, there are some related projects for the Java code, but the setup for those are fairly basic right now (we may do more later though). The C++ projects do the following:
- Builds everything with options for whether to reconfigure, do a clean build, or use a fresh checkout
- Optionally builds and runs all tests
- Optionally runs all tests using Valgrind’s memcheck
- Runs cppcheck
- Generates doxygen documentation
- Publishes reports: unit tests, valgrind, cppcheck, compiler warnings, SLOC, open tasks, and code coverage (using gcov, gcovr, and the cobertura plugin)
- Deploys code nightly or on demand to a test environment and a package repository
Everything is configurable for automatic builds and optional for on demand builds. Underneath, there’s a bash script that controls much of this, which farther depends on our build system, which uses automake and autoconf along with custom bash scripts.
We started using Hudson (at the time) because that’s what the Java guys were using and we just wanted nightly builds. Since then, we’ve added a lot more and continue to add more. In some ways Hudson is great, but certainly isn’t ideal.
I’ve looked at other solutions and the only one that looks like it could be a replacement is buildbot. Would buildbot be better for this situation? Is the investment worth it since we’re already using Hudson? Why?
EDIT: Someone asked why I haven’t found Hudson/Jenkins to be ideal. The short answer is that everything can be improved. I’m simply wondering if Jenkins is the best current solution for my use case or whether there is something better (buildbot?) that would be easier to maintain in the long run even as new requirements come up.
3 Solutions collect form web for “buildbot vs hudson/jenkins for C++ continuous integration”
Both are open source projects, but you do not need to change buildbot code to “extend” it, it is actually quite easy to import your own packages in its configuration in which you can sub-class most of the features with your own additions. Examples: your own compilation or test code, some parsing of outputs/errors to be given to the next steps, your own formating of alert emails etc. there are lots of possibilities.
Generally I would say that buildbot is the most “general purpose” automatic builds tools. Jenkins however might be the best related to running tests, especially for parsing and presenting results in nice ways (results, details, charts.. some clicks away), things that buildbot does not do “out-of-the-box”. I’m actually thinking of using both to have sexier test result pages.. 🙂
Also as a rule of thumb it should not be difficult to create a new tool’s config: if the specification of what to do (configs, builds, tests) is too hard to switch from one tool to another, it is a (bad) sign that not enough configuration scripts are moved to the sources. Buildbot (or Jenkins) should only call simple commands. If it is simple to run tests, then developers will do it as well and this will improve the success rate, whereas if only the continuous integration system runs the tests, you will be running after it to fix the new code failures, and will loose its non-regression value, just my 0.02€ 🙂
Hope it’ll help.
There are many other solutions out there, besides Jenkins/Hudson/BuildBot:
- TeamCity by Jetbrains
- Bamboo by Atlassian
- Go by Thoughtworks
- Cruise Control
- OpenMake Meister
The specifics about what you are doing are not so important, in fact, as long as the agents (aka nodes) that you are doing them on support those tasks.
The beauty of a CI server is noticing when the build changes to trigger a new build (and test), publish the artifacts, and publish test results.
When you compare CI tools like those we mentioned, consider features like the usability of its interface, how easy is branching (and features it might offer like automatic merging), notifications (like XMPP/jabber), or an information-radiator (like hooking up a monitor to always show status). Product support is another thing to consider – Jenkins’ support is only as good as who is responding to community questions at the time you have questions.
My personal favorite is Bamboo, but it comes with a license fee.
The ‘result integration’ is also in jenkins/hudson, and you can relatively easily capture build products without having to ‘copy them elsewhere’.
For our instance, the coverage reports and unit test metrics and javadoc for the java code is all integrated. For our C++ code, the plugins are a little lacking, but you can still get most of it.
we ran buildbot since pre 0.7, and are now running 0.8 and are only now seeing any real reason to switch, as buildbot 0.8 forgot about windows slaves for an extended period of time and the support was pretty poor.