I work in a team that is constantly faced with the challenge of getting features approved for release into production. This is largely because the business is very fast moving, and business priorities change often. As priorities change, the focus of the business shifts from feature to feature, so resources for testing and approving features can be scarce. Consequently, our trunk codeline contains approved and unapproved features. This becomes a problem during our releases, because unapproved features have to be removed from trunk, which as most developers would know, is a painful process that results in dozens of conflicts.
In this blog I will present a solution to minimise the problems surrounding unapproved features in the codeline at the time of a release. This solution involves having a separate branch that only contains features that have been completed and approved by the business. I will also contrast this approach with popular alternatives like Feature Branching and Feature Toggling.
Original Approach
Prior to this solution, commits for unapproved features needed to be taken out of trunk in order to ensure that a software release only contained features that have been approved. This task was often done by a single member of the team. It involved having to revert commits for unapproved features to remove them from trunk. An alternative was to revert trunk to the state it was in the last time a release was done, then merging approved features in one at a time. Either approach is cumbersome and error prone.
Moreover, the UAT environment would be frozen until after the software release, so no one was allowed to bring code into it. After a software release, trunk would then have to be reverted back to the state it was in before unapproved features were taken out.
Problem
This approach worked well for a small team of four at the time. As the team grew to double and eventually triple the original size, this approach was less effective. We were managing around a dozen approved and unapproved features in trunk, instead of the initial two or three. This started to introduce a lot of merge conflicts. The UAT freeze also meant that unapproved features could not undergo UAT during that time. A new solution was necessary.
Solution
We solved this problem using an approach we call Approval Branching. The main idea is that once a feature has been completed, tested, and approved for release, it is then merged from trunk into a separate approved branch. This approved branch then only contains the current production code plus the new features that have been approved. A new execution environment is set up to test the approved branch. Once the code in the execution environment is verified to be stable, we deploy the code from this branch into production.
It is important to note that UAT and approval by the business is done in trunk and its execution environment. Testing in the execution environment of the approved branch only consists of automated Selenium tests to verify a successful merge and also to test for regression. An illustration of our development workflow can be seen in the image below.
Software development workflow using an approved branch
In the illustration above, the approved branch is initially created from the version of trunk that is last released into production. This only needs to be done once to create the approved branch and this branch now runs parallel to trunk. Features F1, F2, and F3 are being worked on at the same time. By the time F1 is completed and approved, trunk also contains F2. The code changes for F1 are then merged into branch. F3 is completed and approved shortly after, and subsequently merged into branch. The next software release comes close and F2 is still not approved. However, this is not an issue because the software release is done from the approved branch, which unlike trunk, has no code from F2. After the software release, F2 is approved and gets merged into branch, ready for the upcoming software release.
Advantages
The first advantage of Approval Branching is that developers do not need to worry about a piece of unapproved or unfinished code sneaking into production. This is a consequence of releasing from the approved branch instead of trunk. As a result, developers can continuously integrate and test their code against trunk during development.
Secondly, Approval Branching does not make the development workflow much more cumbersome at all. The extra work that needs to be done is mainly in merging to this new branch, and testing in its execution environment.
Finally, the UAT environment is not frozen during the time of a release. This is because of the existence of a second execution environment specifically used for this approved branch. Developers can still develop against the UAT environment, and business owners can still test and approve features on it during the time of a release.
Considerations
Several processes have to be put in place to make this work. First of all, in order to simplify the merging process (from trunk to approved branch), every commit for a particular feature must be labelled as such, with a feature number, in the commit log. This allows developers to identify all commits for a given feature that need to be merged into the approved branch for the next software release. A way to enforce this is to use a pre-commit hook. The VCS can then check to make sure that the first line of every commit log begins in a given format, e.g. “PROJECT-A-###”, where ### is a three digit number that identifies the feature.
Also, the same people who work on a feature must be the ones to do the merge into branch to minimise the risk of a bad merge. A feature is only considered complete once it has been merged and verified by the developer in the execution environment for the approved branch. This is a better approach than having a release manager who specialises in merging approved features into the approved branch. It is much easier and takes less time for the developer of a particular feature to merge his work into branch, than it is for a release manager who is unfamiliar with the particular feature to do it.
Lastly, in order to reduce the amount of extra testing that needs to be done as a result of this new execution environment, automated Selenium tests are triggered to test for regression and a successful merge.
Concerns
Readers might be concerned with this approach because it involves branching and merging. However, it is important to note that we rarely encounter merge conflicts, and when we do, they are usually for configuration files. These merges are fast and easy to resolve.
This approach does introduce an additional execution environment for the approved branch that requires extra maintenance and testing, which I believe is the biggest drawback of Approval Branching. Our team mitigates this issue by using a continuous integration tool to bring code from the approved branch into this new execution environment.
Comparison to Alternatives
Martin Fowler has discussed Feature Branching and Feature Toggling in recent years. These are both widely used techniques for release management, so naturally our team considered these alternatives before deciding on Approval Branching. What we found is that these alternatives do not work as well as Approval Branching in practice.
Feature Branching
Feature Branching encourages a developer to have a branch for every feature that the developer is working on. Once a feature has been completed, its corresponding branch is then merged into trunk. This keeps trunk stable with features that have been fully developed.
However, Feature Branching does not solve the issue of unapproved features being on trunk. Once a feature has been developed, it is merged into trunk, where UAT is carried out. What happens then when a feature is not approved?
In addition, it is difficult to get feedback from the business during the development of a feature because feature branches do not have dedicated execution environments. If a feature needs to be reviewed during its development, this branch has to be deployed to an existing execution environment (one perhaps already in use for trunk). This means new features in trunk cannot be tested, and the execution environment is essentially frozen to other members of the team.
Feature Toggling
Feature Toggling involves implementing a switch to turn on/off the UI elements of certain features. If a feature is approved, the UI elements are switched on. Otherwise, they are hidden from public view. In order to reduce clutter in code, developers are encouraged to remove switches for approved features from code.
We decided against this approach for several reasons. The first reason is developers need to write extra code to switch on/off public-facing UI elements. It is unlikely that someone would spend time to go back and remove a switch from a feature that has already gone live. Therefore, the clutter in the code will most likely stick around until it becomes too dangerous to be removed.
Feature Toggling is also prone to errors. A switch can be left out of certain UI elements, causing unapproved or incomplete features to sneak into production, and negatively affect the end user experience. In effect, this means that extra testing needs to be put in place to test for the behaviour of the system with a feature switched on or off.
Conclusion
Release management is a complex issue that many developers have to deal with at one point of their career. Feature Branching is unfit for our purposes, and Feature Toggling is complicated and relies too much on developers doing perfect things in a perfect world. Approval Branching eliminates many of these concerns, while preserving the current workflow of the development team. Approval Branching is powerful because it acts as an extension, rather than a transformation, of an existing process.
