Bodhisattva in Training

September 16, 2007 Continuous Integration

Consequences of Pipeline Structure

There are two general ways in which you can structure a build pipeline: one cuts across the product and the other across validation.

I was recently asked about breaking up the build by having a build hierarchy that mirrored the product’s package dependency tree. I have seen this done, and done it myself, in a very coarse way, dividing along lines strongly marked both by organization and architecture. There is a even a well known pattern on this: Named Stable Bases. It should be made clear that this is a form of deferred integration. Yes, yes, the ill consequences of deferred integration are the very things that continuous integration is working to alleviate. Just because deferred integration has been abused in the past is no good reason not to wield it wisely now. So lets take a look at how it alters the dynamics of a build system. To show this I have created a loop diagram, see this post for more details on loop diagrams and remember that the “s” means varies the same and “o” means varies the opposite.


The entire product is not built, even compiled, all at one time; it is spread out over time. Many times there can be a significant delay between when a dependency is built and when a dependant consumes that new build. This breaking up of the build has the effect of decreasing the size of the Codeline which ultimately shows the effect of increasing the rate of change to the Codeline. This is a short sighted view, remember that we are building a whole product and that we have deferred some integration. The key here is some, not all. So the question is will the things effected positively out weigh the deferred integration? I have illustrated the forces at play here in a well know pattern or archetype in Systems Thinking named Fixes That Fail.

Fixes That Fail (^)

The Fixes That Fail structure consists of a balancing loop and a reinforcing loop. These two loops interact in such a way that the desired result initially produced by the balancing loop is, after some delay, offset by the actions of the reinforcing loop.

The internal balancing loop operates in the standard balancing loop fashion. The Action that influences the migration of the Current State also influences, after some delay, some Unintended Consequences. These Unintended Consequences subsequently impede the migration of the Current State in the intended direction.

One of the dangers I see with this approach is how it shifts the focus away from the product and the overall build systems to the product components and their smaller builds. It is more difficult to see the big picture and foresee the repercussions of actions in the small on the large. This type of build pipeline shifts the delay of feedback from an individual integration duration to the push or pull between individual builds. For example there is an uncomfortable delay in getting a developer feedback when the build of an entire product takes 45 minutes. One might feel a good solution is to break up this single build into 3 builds one for the common libs, one for the server, and one for the client. Now when a developer of the client submits to the build they get feedback in 8 minutes. Here is the deception of this model, the overwhelming majority of the time the client build is working against old versions of the common libs and server. The delay is getting the build output of the common libs and server integrated into the client build.

In my experience most people start down the path of splitting a build in this fashion by first seeing a waist in recompiling packages that don’t need to be recompiled. This is a valid observation and fruitful optimization, and I think that in most cases division of build process to realize this benefit yields a net negative in consequences. I have only seen this structure work well as a means to compensate for organizational issues. For example when the web front end team has little contact with the back end service team. The web front end team may experience interruptions due to changes in the back end that impede progress. One means to compensate for this is to accept regular stable deliveries of the back end. This allows for uninterrupted progress, flow, and regular planned integrations to the new version of the back end.

I have experienced great success with division of the build across validation as apposed to across the product. In the past I have referred to this as creating developer, release, tester, and customer facing builds. Let’s generalize that a little to: role facing builds. Before I explain remember that the desired state is rapid feedback and the gap is perceived delay in feedback. When we take into account the different roles on the team we realize that they each have different desires. The developers are accepting of less comprehensive feedback in exchange for quicker execution times. The testers in contrast are accepting of longer execution times in exchange for more comprehensive feedback. The developers are not interested in creation of an installer, verses everyone else is extremely interested in an installer. If you continue down this path of inquiry you will have a set of values that will help you shape a build pipeline. I bet that you would find only the developers perceive a delay in feedback. Creating role facing builds can be quite liberating in what you can do and what you don’t need to do in a build. It is the don’t need to do that has the greatest affect on the developer facing build. You don’t need to do a clean build, an incremental or dirty build can be very quick. You don’t need to execute long running tests. You don’t need to create an installer.

This is not a perfect solution, there is no such thing. So what about the negative consequences? A successful build from a developer facing build can give a developer the illusion that all is well. They need to remain engaged and interested in the results of the cascaded builds. I have not seen any other generic issues with this approach, here is an example of an issue with how it is implemented: It is important that the developer be able to easily execute all parts of any build in the system that can cause a failure and or create build artifacts. If for example a developer can not run a test that is causing a build to fail they will not be very likely to jump right on it. There will be negatives in what ever implementation you conceive of. You need to be alert to them as you will not be able to foresee them all. There are many ways in which to mitigate and or alleviate there impact.

Creating a loop diagram of your build system can be a tremendous help in this regard. Here is a picture of a generic build system to help illustrate the value:


Here is a link to the free software that I drew this with and the diagram itself:


Leave a comment