Have you ever worked somewhere where they deployed once a quarter? I have. It sucks and it’s super risky. On the other hand, I’ve been at places where we push to production over 1000+ times a week. “But we have 75 people on the call and they’re all paying attention”. Yeah, OK. I’ve been on these and I’ve heard people sleeping. Midnight calls suck and sleep deprived people who are deploying large amounts of code with a lot of steps manually is RIPE for error. Making mistakes happen and “short deployments” turn into hours and you get delayed even further. Now imagine, your archaic company has a deployment window and you missed it and you have to go the next quarter. Congrats, you’re now going to take 6 months to get some features into the hands of customers. Anyways, I’m here to discuss manual lengthy deployments and why they’re risky. I’m not saying change your process right away (you can contact me for that). However, start thinking of the ramifications of stale code and manual deployment risks.
Why are manual deployments risky?
Manual deployments involve several steps that humans must oversee and execute. This process is typically slower, more error-prone, and riskier than automated deployments for a variety of reasons:
- Human Error: People can make mistakes. With many steps and people involved, there are more opportunities for human error, such as entering incorrect data, missing steps, or misunderstanding instructions. These errors can lead to issues ranging from delays to serious production outages or even security vulnerabilities.
- Reproducibility: Manually executed deployments may not always be consistent. Slight variations in the execution of the process by different team members can lead to different outcomes, which can cause problems when trying to replicate or troubleshoot issues.
- Knowledge Transfer: If only a few people know how to deploy the application, this can create a single point of failure. What if they leave the company, get sick, or are otherwise unavailable? This can cause delays and problems with maintaining the application.
- Slower Reaction Time: Manual processes typically take longer than automated ones. If there’s an issue that requires a new deployment, like a critical bug fix or security patch, the delay in getting the deployment out can have significant consequences.
- Scalability: As the system grows, the number of steps and complexity typically increases. The time required for manual deployments grows with it. Automation allows for faster, more scalable processes.
- Traceability and Auditing: Manual processes often lack thorough logging, making it difficult to trace back steps and understand what was done when problems arise. This can also present challenges for auditing and compliance.
- Cost: Manual processes require more human time and therefore cost more. Over time, these costs can significantly add up.
- Context Switching: Engineers involved in manual deployment have to shift their focus from their primary work, which can disrupt their workflow and decrease productivity.
How do I go from manual to automated process?
Setting up a CI/CD (Continuous Integration/Continuous Deployment) pipeline from a manual process involves various steps. It’s actually really difficult to swap to automated deployments due to various reasons. Here’s are some reasons why you might face these issues.
- Loss of Control: Some might fear losing the hands-on control that manual processes offer. They might believe that automation is a “black box” that makes it difficult to intervene if things go wrong. It’s actually inherently less risky releasing small, frequent code rather than one big push every 6 months.
- Complexity: Setting up automated deployment systems can be complex. This can make the transition intimidating, particularly for smaller teams without dedicated DevOps understanding. Generally, a more likely scenario would be somewhere in an organization, you’d try out these new deployment pipelines and understand how they work before rolling them out elsewhere as you build knowledge.
- Risk of Failure: There can be concerns about the risk of something going wrong during the transition period, potentially leading to downtime or other negative impacts on production environments. This is normal. This goes back to dora metrics, cycle time, MTTR and more. Teams aren’t elite off the bat and need to work on things. How you treat risk at your organization is important.
- Trust in the Automation Tools: Some might have concerns about the reliability or robustness of automation tools. They might worry about bugs in the tools themselves leading to problems. Ie, how do I know this really works?!
- Cost: There can be concerns about the costs of implementing automation, particularly for smaller teams or organizations. These costs can include not only the direct costs of the tools but also the time and resources needed to set up and maintain the automation system. CI/CD is just one part of the large picture on devops culture. It’s important to always continuously improve and research.
- Lack of Knowledge: If team members are not familiar with automation tools and practices, they might hesitate to adopt them. This can be addressed through training and gradual introduction of automation. For example, having 1 team member who is a “Devops” expert will lead to a lot of problems across the organization. Devops is a culture and a mindset on how we deliver software. There’s also definitions of what elite software teams look like so that helps.
- Fear of Job Loss: Some may fear that automation will make their roles redundant. In reality, while automation does change the nature of some roles, it often frees up people to focus on higher-value tasks rather than repetitive, manual work.
OK, you’ve heard of some of the resistance to a CI/CD pipeline. Let’s get into a theoretical setup. Some way easier than others. For starters, I’d probably use an all in one tool with a test team to see if you can understand the process and be champions across the organization.
Here’s a generalized approach on setting up CI/CD:
- Identify the Current Process: Start by understanding the existing manual process. Document each step, noting any dependencies or manual interventions. Also, identify any potential bottlenecks or areas of frequent failure.
- Choose Your Tools: There are various CI/CD tools available, such as Jenkins, CircleCI, GitLab CI, GitHub Actions, Travis CI, and more. Your choice will depend on factors like your existing tech stack, budget, and specific needs. For instance, if your source code is already on GitHub, GitHub Actions could be a natural choice. If you’re completely new, Gitlab can be a great all in one tool. It really depends on where you are in your journey.
- Create the Build Process: Set up the process to automatically build your code whenever changes are committed to your repository. This includes tasks like compiling code, running preprocessors etc.
- Automate Testing: Integrate automated testing into the pipeline. Unit tests, integration tests, functional tests, and other types of tests should run automatically when code is committed. Only builds that pass all tests should move on to the next stage. You may not be able to get great coverage early on and that’s OK. Understand the concept of it.
- Automate Deployment: Set up automatic deployment to a staging environment for builds that pass all tests. Depending on your setup, this might involve tasks like copying files to a server, running database migrations, or invoking deployment scripts.
- Approval for Production: For CD, you may want to automatically deploy to production after successful tests, or you might have a manual approval step. The latter can be useful if you want a final review or if there are business considerations about when to deploy.
- Rollbacks: Implement a process for quickly and easily rolling back deployments that cause issues in production. This might be as simple as redeploying the previous version of your code, or it might involve more complex database rollbacks.
- Monitoring and Logging: Your pipeline should automatically collect logs and monitor your applications. This will help you detect and troubleshoot issues quickly.
- Iterate and Improve: Once you have a basic CI/CD pipeline in place, you can keep refining it. You might add additional types of tests, improve your monitoring, or streamline your process further.
Transitioning from manual processes to a CI/CD pipeline involves an upfront investment of time and resources, but it can greatly pay off in terms of faster, more reliable deployments, better code quality, and more productive teams.
Wait, we just go straight to production? What about reviews!?
The goal of Continuous Integration and Continuous Deployment (CI/CD) is to automate the software release process, from integration and testing phases to deployment. With a mature CI/CD setup, the idea is to ensure that the code is always in a releasable state. While CI/CD enables faster, safer, and more frequent deployments, it’s worth noting that a fully automated pipeline straight to production may not be suitable for every organization or project. You might choose a Continuous Delivery model where every change goes through the pipeline and is prepared for release, but final deployment to production still requires a manual approval step. This approach offers a balance of speed and control, allowing for human oversight when needed.
Here’s why we often deploy directly to production with CI/CD:
- Faster Time to Market: CI/CD allows teams to deploy updates more quickly and frequently. Instead of waiting for a “release day” to roll out new features or bug fixes, they can be pushed to production as soon as they are ready. This accelerates the delivery of value to the end-user.
- Improved Quality: With CI/CD, every code change triggers an automated pipeline of build and test processes. This means bugs and issues are caught and fixed more quickly, resulting in improved software quality.
- Increased Productivity: Automation frees developers from time-consuming manual tasks such as integration and deployment, allowing them to focus more on writing code and less on the mechanics of getting it into production.
- Reduced Risk: Deploying smaller changes more frequently reduces the risk of each deployment. If an issue does occur, it’s usually easier to diagnose and fix because you have a smaller change set to consider.
- Faster Feedback Loop: By deploying directly to production, teams can gather feedback from end users more quickly, which can help guide future development efforts.
Closing
So there you have it. From my days in banking releasing stuff at 5am with 40 people on a call it only to be pushed later in the quarter; to deploying code 1000s of times per month at a tech company. You may or may not be ready yet for ci/cd but there’s steps you can take to reduce risk and deploy more often. Feel free to contact me if you have any questions or comments.