Episode 22 - 5 tips to taking the fear out of automation projects

Devops Mastery

A lot of people new to automation are very nervous about letting a script or program like Chef change a production system. That fear can really slow the advance of a DevOps movement. But ignoring those people's fears is not the answer. So here are some tips and tricks to get everyone, including yourself, to be more confident about writing world changing scripts. Whenever possible create virtual environments. The use of virtual machines or cloud servers gives you considerably more options and abilities than bare metal hardware. Simply the ability to snapshot a server and revert back to it in minutes is a game changing concept. It let's you run any automation from beginning to end virtually without fear. I say virtually only because you occasionally run scripts that affect systems that cannot easily be virtualized. This is now more the exception than the rule. If you take a snapshot of the server, run the automation, find a mistake/error/bug you simply revert the server back to the starting point, fix the problem and try again. The only thing you have lost is the time it takes to run the script. This might be a lot of time but still less than having to rebuild the environment between each test run. If you have the luxury of multiple environments to use you can and should follow the same procedure used in the development of the script for future deployments. So you just take the script that worked in Dev and run it in Test after taking a snapshot there. Then rinse and repeat what you did all the way to production. Test it over and over and over again. It sounds crazy but I now run the completely new scripts I am ready to bless for migration to the next environment multiple times in each environment. This gives me the confidence that they really are working. I want to know I didn't do something weird that made the script work when I fixed the bug. Once the script runs cleanly multiple times I am all ready to go and I move it to the next tier. Ask someone to do a peer review of the script. This is just a good practice in general. When possible I suggest walking the person through the script and letting them ask questions the first time through. This has often turned up bugs and other issues all on it's own. Having to explain something uses a different part of your brain and opens your mind up to see the problems or possible problems. Ask someone else to run the scripts for you. I normally do this in a Dev/Test environment. I do this for two reasons 1) Validate my documentation 2) validate the procedure I am doing. Often my teammates have asked questions that point out things I forgot to account for or code in. They also tend to be less stressed than if they are deploying to a production environment. Doing this earlier is better. Repeating it when you have to make a change makes you a better programer/scripter. Your teammate finding that one thing that would have driven you nuts during a deploy is, in a word, priceless. Communicate what it is you are trying to automate. Be as specific as possible in the communications. Making everyone involved aware of what it is you are doing goes a long way towards building their trust. It is the most basic way to be inclusive in your process. Do your best not to make it sound like you are asking for permission to do the automation. You are doing this to make the environment better and no one should ever question that. This doesn't mean that you shouldn't be open to feedback. You should actually be ready for it because it will happen. While everyone in any given IT organization may not be expert communicators you need to take all feedback without feeling persecuted. Remember this is just as scary for them as it is for you at the beginning. So they may need time to adjust and gain confidence also. The worst thing you can do is start or continue a flame war about your automation process. Respond respectfully and with clarifications about things like how you are planning to test, why you chose to do it the way you have, and that it is an always evolving process. What you start out with in the first round of automating anything will almost never be what it is by it's Nth revision. Finally, don't let the communications process stop or stall your automation of anything completely. I normally do this by focusing on the fact that this is just one step in an every progressing path of enhancements to your current processes. So to summarize start using virtualization, test and retest, do peer reviews, have someone validate your scripts and documentation, and communicate early and often. With each piece of automation that follows you will refine how these things need to happen in your organization to get the highest level of trust. Finally remember that what works for one application or service will almost always need to be tweaked for the next one's specific needs.