In my interview with David, we created a process map of software development at NuoDB. After these interviews, I like to suggest how the processes we described could be improved. Unfortunately, in this case, I can find no clear suggestions for improvement. But that doesn’t mean there is nothing to talk about! There are interesting things happening in each of the software development subprocesses.
Work definition
OK, maybe there is nothing interesting to talk about here. The product managers worked closely with the sales team to prioritize features. Sometimes you had to drop everything to add a feature to close a deal. Serious bugs were handled urgently. Prioritized tasks were estimated and placed into a release cycle. All pretty standard stuff.
Work preparation
NuoDB relied on a rigorous design review process in order to make software changes properly. This was a weak point in the development process because it relied on having engineers being heavily invested. Engineers need to care a lot in order to make such detailed evaluations of designs. But it makes sense that the design needed to be performed carefully because they could not perform rollbacks when things went wrong (more on this later).
Software development
NuoDB had an intense code review process where all engineers could comment on any code review. Sometimes it could take many weeks for a change to be approved. Sometimes changes would not be approved and the work would return to the design stage. This code review process kept bugs out of the system but certainly slowed down development. And it relied on having engineers that were invested enough to provide a high level of scrutiny on the code reviews.
Software release
I am surprised that they deployed to on-premise installations, this is not a typical deployment strategy anymore. It makes sense that those releases would be infrequent (every six months) because they needed the release to be performed by the customer on their machines. It also makes sense that these deployments could not have downtime-databases cannot easily have downtime because information might be missed.
Together those requirements added a lot of complexity. Because of the on-premise installations with infrequent upgrades, NuoDB needed to maintain multiple versions of the software. And NuoDB could not really perform rollbacks because upgrades were performed by customers on their own equipment. Rollback logic would also be quite complex due to the “no-downtime” requirement. This complex and difficult deployment drove the need for rigorous design and code review processes.
Conclusions
Overall NuoDB had a healthy software development process. If I need to make some critiques I would definitely point to the reliance on invested and detail-oriented software engineers. It is not always easy to find these people, and without them, the company would not be able to continue its software development process. Some engineers do not enjoy the slower and more deliberate pace of development. And junior engineers might struggle to ramp up in a slow-moving environment.
I do not see a clear solution to this problem. NuoDB had automated and manual testing suites to make sure their software was not broken, but automated tests are never enough to guarantee that things will work. Normally this gap is bridged by rolling back bad changes. For many services when a bad change is deployed, a rollback can quickly and easily undo the change. But when you need to be absolutely sure that the software will work, you need to rely on software engineers to be very careful and deliberate. I am impressed that NuoDB was able to create such a focused software engineering culture, but it takes a lot of effort to maintain such a culture.
Leave a Reply