One cloudy day deep in despair I ascended Mt. Si seeking answers and thereupon I found an Android tablet inscribed by the finger of Hephaestus. That sacred tablet contained the following three decrees of software release in a read-only file.
1. Thou shalt use the following terms unambiguously
Environment: A complete and working version of the service. A single binary can be deployed to an environment at any given time.
Binary: A collection of assets that can be bundled together and deployed to an environment to run a service. A binary can contain uncompiled code so use this term even when programming with interpreted languages like Python or Javascript.
Binary Deployment: A binary can be deployed to an environment to run a service.
Feature Deployment: You must not confuse a feature deployment with a binary deployment. You can deploy a binary containing a new feature behind a flag that is set to false. The binary release can proceed and then the new feature can be deployed later by flipping the flag. Thus the feature deployment is decoupled from the binary deployment.
Promotion: When a binary that is deployed in Environment A is deployed to Environment B you shall instead say that the binary is promoted from Environment A to Environment B.
2. Thou shalt deploy to the following three environments
- A dev environment
- A staging environment
- A production environment
New binaries can be built and deployed to the dev environment every hour. On Mondays and Wednesdays, the most recently deployed dev binary can be promoted to the staging environment. On Tuesdays and Thursdays, the most recently deployed staging binary can be promoted to the production environment.
End-users shall only access the production environment.
3. Thou shalt document the following manual processes
How to roll back an environment: If a bad binary is deployed to an environment you roll back by redeploying an old binary to that environment.
If you deploy a bad binary to the staging environment you will need to find the binary that was deployed to staging two days ago and redeploy it.
Thou shalt not roll back twice in a row.
How to roll forward an environment: A roll back will not always fix your service, sometimes instead you may need to roll forward. A roll forward is a promotion of a binary that is outside the normal schedule.
If you deploy a bad binary to the staging environment, but a fix is available in the most recent binary deployed to the dev environment then you can perform a roll forward by deploying the most recent binary deployed to the dev environment to the staging environment.
How to cherry-pick a fix into a binary: Sometimes you need to deploy a fix very urgently. In such an unenviable situation you need to be able to rebuild a binary with a specific fix included and deploy it immediately. This is called a cherry pick.
If you have a serious issue in your production environment and a single line of code will fix it you may want to build a new version of the production binary with that one line of code changed (cherry pick the change) and then immediately deploy that binary to production.
How to block binaries from being promoted: Sometimes you need to block promotions to an environment.
If you perform a roll back on the staging environment, you may need to block promotions from dev to staging for a short period to make sure that the bad change that was rolled back does not get deployed again.
Leave a Reply