Last week i had the awesome honour of being at a tech talk on Continuous Delivery by “the great” Martin Fowler, Neal Ford and Evan Bottcher, put on by Thoughtworks here in Sydney. I have spent the last two years of my life evangelizing on the benefits of Continuous Integration and Continuous Deployment to anyone who would hear me all on my own terms and entirely self taught in the dark. I have selected snippets and ideas from other peoples documented ways on how they achieve Continuous Deployment, but the experience of listening to pioneers in this space was a truly awesome experience.
I took away from it a lot of little golden nuggets that reinforced everything i had experienced first hand, and i thought I'd share some of the things that were said.
I’d like to let you know that i refer to the process of going live with a site or to staging with a site in an automated fashion as “Continuous Deployment”. Martin and the gang refer to this as “Continuous Delivery” and explain why they use their term and no mine.
They explain the major difference as they see it is that deploying is not always considered as releasing software into a live environment. Feel free to use either, however for the purposes of this post, know that i refer to it in both forms, but i am talking about the same thing…
Enjoy :-)
“… If it hurts, do it more often …”
Martin Fowler
One of the greatest things that Continuous Integration in general, and Continuous Deployment/Delivery adds to, above all else is allowing everyone working on a project to get visibility over a projects gremlins – its true list of dependencies, brittleness and architectural weak points. As i mentioned in my previous post on Continuous Deployment, the ability at the earliest possible point in time to find any weak points in your projects ability to flexibly deploy at the drop of a hat is priceless. Once you have experienced this you simply cannot turn back.
“… Flickr deploys sometimes up to 25 times a day …”
Martin Fowler
Flickr attributes its ability to deploy new features in such a rapid fashion a large part of its success as a dot com. This is mostly because they are in such a confident state of mind about where the project is and what has changed since it was last deployed, that they are not afraid of pushing changes onto the site all the time. They have made a lot of good architectural decisions that work well with their CI setup that we can all take something away from. They add new features to the site and allow them to be switched on with configuration flags so they can be easily rolled out, rolled back and tested in different configurations on development, staging and live environments. They incrementally roll out database schema changes, sometime duplicating columns, and writing code in their data layer that passively migrates data to new schema layouts. This means they can continue to innovate, try new things and stay ahead of the curve in a very aggressive fashion – a major part of this comes from thinking about their deployment strategy as part of their project planning from day one.
“…Continuous Delivery is not Continuous Integration. Continuous Delivery is being in the position to ship your product whenever you want, day or night…”
Neal Ford
Continuous Integration has been growing steam for quite a while. Since the inception of things like CruiseControl and TeamCity they have become even more common place in agile development teams. Continuous Delivery is not Continuous Integration, because it takes the idea one step further. One of the points they made really clear on this is the fact that knowing your project builds doesn’t deliver the same confidence that Continuous Delivery does, as you are simply moving the fear factor further down the projects life cycle – in most cases to the last week before you ship or go live with a project. This is not how things should be, as you are going to the extra effort of automating, without the pay off.
“… Manual releases are like creating magical one of a kind snow flakes. They are not repeatable and hard to recover to in the case of absolute infrastructure failure – Because of this they should absolutely not used in modern development practices…”
Martin Fowler
When you leave your deployment to the last minute or last week of development, you are always in for an uphill battle to ship your product. This leads to bad design choices, last minute changes and patch fixes on both your software end AND your operating environment (updates, service packs etc). Because of this the story goes that you are often going live over a weekend where everyone stays back and hopefully by Sunday evening( or early monday morning!!!) everything is working, but because of this a lot of the steps to pushing it live are once off, non-documented, “i think we were running it under service pack 1” style actions. This leaves you with a very brittle, hard to replicate system, and in the extreme case that history has shown us about some companies, absolute hardware and infrastructure failure can simply mean they have to close shop. You can avoid all this by using Continuous Delivery from day one. If your US servers die and you lose absolutely everything, you simply change your deployment destination to new hardware in your new hosting providers data centre and push the button – this is major insurance for your product and makes the business case for Continuous delivery very strong.
“… Computers are designed to do simple repetitive tasks. The second you have humans doing repetitive tasks, all the computers get together late at night and laugh at you…”
Neal Ford
Developers are hired to write code, be creative, build castles in the sky. Human beings are designed with an imagination that, in the development industry, allows them to solve complex problems, deliver elegant solutions and command computers to great ends – the second you make a programmer do something over and over again that can be automated, you raise your risk for mistakes, and stop them from doing other, possibly great things. You are wasting their ability as hired professionals to the detriment of the project at hand – they could be adding new features or innovating in ways that less development time simply can’t achieve.
“… Source control is number one in both configuration and logic storage. A new developer needs to be able to have everything in a state where they can load it up in 1 command, and deploy it with another…”
Evan Bottcher
One of the greatest things about Continuous Delivery, is it forces everything a project needs to exist into source control in a very tight ecosystem. Have a database? the schema is in source control and can be rebuilt anytime of the day. Have some third party binary dependencies? they are included with the project in source control and are included in the build.
Your build server needs to have everything on hand to deploy the project, so everything goes in your source control repository – you create this wonderful centralized place where everything that a project consists of can be loaded from one place and run, because from day 1 you’ve had your build server in mind. Neal Ford went on to add that he has worked a number of places where on his first day he has put some code into production using Continuous Delivery and because of this it made him careful about how he made his change, as he knew it would go live and he didn’t want to break the build. This kind of fluid development while keeping care for quality cannot be achieved without continuous delivery
“… In 2011 is software that cannot be automated, is broken. Get new software…”
Neal Ford
Because the agile way of thinking has been around for a number of years and continuous integration has been a growing phenomenon, there are always a number of choices when it comes to using third party software to achieve something, hell there will usually be a free open source version of something that does what you want. Whether it be automating your build with tools like TeamCity, Cruise Control or Bamboo. Your deployment with Web Deploy, scripted ftp clients and WebDAV. Or taking database schema dumps with tools like RedGate’s SQL source control or SQL DB Control, someone has written something to do it.
Continuous Integration and Continuous Delivery require the use of tools that accept command line execution and scripting capabilities. If your software of choice does not support this, get new software – this is not an excuse to not add Continuous Deployment/Delivery to your project and software packages that don’t take this into account in this day and age, are not worthy of your time.
“… Continuous Delivery is all about reducing risk – if you ever need something to sell the idea of Continuous Delivery to management, this is a walk in the park – you are able to minimise risk to your project with this one thing more than anything else in your project lifecycle…”
Evan Bottcher
I’ve spoken to a lot of people about Continuous Delivery/Deployment over the years and a lot of the time i hear that the amount of time that it takes to implement it into a project is a cost their managers are simply not willing to allow them to incur.
This is ludicrous.
If you work out the amount of time you save in the long run by having a good Continuous Delivery setup, and couple that with the risk management benefits that finding bugs earlier, being able to guarantee delivery dates and keep staff happy by allowing them to go home at a normal hour of the day, it’s a simple no brainer – it’s easy to sell to management as long as you keep these things in mind.
“… Every time you do something for the third time, automate it. You’ll be doing it a million times…”
Neal Ford
Once you setup your our Continuous Delivery implementation, one of the first things you may ask yourself is what you should be automating. Neal’s advice is a pretty golden rule. Usually the things you think you’ll only do a couple of times end up being the biggest time hogs. Free yourself and you colleagues! automate these things into part of your build and delivery process.
“… You want the release week to be the easiest week of the project life cycle because you’ve done it already 60-70 times…”
Martin Fowler
I mentioned above that one of the benefits from a business point of view is that you actually reduce a developers work load. Staff turnover at a company that always does thing the hard way can be quite high. By adding continuous delivery you allow your staff to have a lot of confidence in the project’s stability which inturn allows them to lead more normal lives outside work. They go home on time, have more time to innovate and add quality to a project and it allows them to go live confidently and without the stress of a looong weekend deploying. A happy developer leads to a good end product.