Continuous Integration is a software development practice where continuous changes and updates in code base are integrated and verified by an automated build scripts using various tools.
Continuous Deployment is also a software development practice whose role is to automatically deploy the code to the specified server and application folder once the code is been integrated successfully.
We will use couple of famous tools to CI and CD my app to any particular environment.
Suppose we have setup 3 different environments (Development, Staging and Production) and all the server are architectured on Amazon Web Services.
As a part of architecture , Our application was deployed on 3 tier architecture (Client Tier, Business Logic Tier and Database Tier) and it was featured with AWS Auto Scaling service which used to help scale our servers depending on the metrics and policies we specified.
So, every time a new feature was developed, we had to manually run the test cases before the code was integrated and deployed later pull the latest code to all the environments.
1) Manually work to run test cases and pull the latest code on all the servers.
2) Since the servers were auto scaled we had to pull the latest code on one server , take image of that server, re configure it with Auto Scaling.
3) Roll back to last successful pull was a painful task.
Though the above challenges are not the blockers but we have to bring up a technique that must save high amount of time and make our life easy with automating all the process from CI to CD.
We choose to use Jenkins as CI tool and AWS Code Deploy as a CD tool
Lets walk through the flow before we implement it all.
When a new code is pushed to a particular GIT repo branch.
1) Jenkins will run the test cases, (Jenkins listening to a particular branch through git web hooks )
2) If the test cases fail. It will notify us and stop the further after build actions.
3) If the test cases are successful , it will go to post build action and trigger aws code deploy.
4) Jenkins will push the latest code in the zip file format to AWS S3 on the account we specify.
5) AWS Code Deploy will pull the zip file in all the Auto Scaled servers that have been mentioned.
6) Once the latest code is copied to the application folder , it will once again run the test cases.
7) If the test cases fail it will roll back the deployment to previous successful revision.
8) If it is successful , it will run post deployment build commands on server and ensure that latest deployment does not fail.
Now lets follow above steps and configure everything from scratch.
1) You have a Git Hub account with admin access and a sample repo.
2) This article is written for Ubuntu 12.04 but the instructions are universal
3) You have AWS User Credentials , Secret Key and Access Key with Power user rights
4) Ive configured Jenkins on my local machine and i have my application server on AWS where the application updates are going to be deployed automatically.
A. CONFIGURE AWS CodeDeploy Application:
I assume you have all the access required to configure AWS CD
1) We need to configure 2 IAM Roles that would be used to boot your ec2 server / access S3 and then for Code Deploy to talk to ec2 server.
We will name them CodeDeploy and EC2CodeDeploy
To create the role go to IAM Service select Roles -> Create New Role
Enter the name CodeDeploy and select Next
Select Amazon EC2 as Role Type and select Next
Under Set Permissions choose Custom Policy and select
Enter the name CodeDeploy and paste the below policy
Select Next and Create Role. This show the list of roles created.
Now we need to select CodeDeploy and select Edit Trust Relationship
Replace the policy with below policy
Policy Trust for CodeDeploy
and Update Trust Policy
2) Now you need to follow the above steps to create EC2CodeDeploy policy
Paste the below Policy
No need to edit the Trust Policy.
3) Now you need to launch Amazon Linux (contains aws cli pre installed) or any other instance, in my case Ubuntu attached with IAM role EC2CodeDeploy and install aws cli with aws code deploy agent.
To Install CodeDeploy agent follow below steps
Considering you have installed aws cli
aws configure : Fill in Secret Key, Access Key and Region (CodeDeploy only supports N.Virginia and Oregon region)
aws s3 cp s3://aws-codedeploy-us-east-1/latest/install . --region us-east-1
chmod +x ./install
sed -i "s/sleep(.*)/sleep(10)/" install
service codedeploy-agent status
Remember that you need to place the appspec.yml file in your application repo.
here is one sample appspec.yml file
- location: scripts/beforecicd.sh
- location: scripts/aftercicd.sh
The beforecicd.sh script files contains all the commands that needs to be done before the code is deployed to the application folr.
In my case I create a folder that will just take a backup of some files.
aftercicd.sh file contains commands that will run some test cases to check if the application is deployed properly.
4) Now we will create AWS CodeDeploy Application
Go to AWS CodeDeploy -> Get Started -> Custom Deployment
This will bring you to Create New Application page.
Fill in the Application Name and Deployment Group Name
The next interesting part is selecting the instances where you want to deploy the application.
In Add Instances part you can search and select your Instances or AutoScaling Group by tag names.
Here we select the instance that we had launched.
Deployment Config , here we can set it to One at a Time since we have only one server for now.
Service Role , here we attach the tole CodeDeploy Role that we had created.
Finally select Create Application.
Here we finish with configuring AWS CodeDeploy
Now next part would be to install and configure Jenkins.
July 1, 2019
DevOps is widely adopted as it has shortened the software and application development life cycle… Continue reading...
January 16, 2019
What is Istio Service Mesh? Istio service mesh provides several capabilities for traffic monitoring, access… Continue reading...
January 7, 2019
What do companies like Amazon, Target, Esty, Netflix, Google and Walmart all have in common?… Continue reading...
December 27, 2018
The Anatomy of Continuous Integration Tools Some of the best continuous integration (CI) tools aren’t… Continue reading...
February 1, 2018
It was not a long time ago when developers were required to know everything about… Continue reading...
December 7, 2017
Large and Mid-size organizations today are trying to act like startups. They outsource software development… Continue reading...
September 27, 2017
If somebody asks you what is the biggest challenge facing your software engineering team right… Continue reading...
July 8, 2016
"DevOps is a slightly misunderstood concept by many. For me DevOps is not a “new… Continue reading...
June 15, 2015
Continuous Integration is a software development practice where continuous changes and updates in code base… Continue reading...
June 9, 2015
Deploying ROR app with Ansible: Ansible is a lightweight, extensible solution for automating your application… Continue reading...
March 25, 2015
Every computing language has its history, strong points and a framework around which it works.… Continue reading...
March 25, 2015
As DevOps culture is being adopted inside IT industries, so does the interest in automation… Continue reading...
February 19, 2015
DevOps is an umbrella concept that smooths out the interaction between development and operations &… Continue reading...
February 17, 2015
"Why, sometimes I've believed as many as six impossible things before breakfast." - Through the… Continue reading...
January 29, 2014
Do you have a compelling product idea? Want to build a new product? Are you… Continue reading...
B. INSTALL JENKINS
1) We have to add the key and source list to apt.
wget -q -O - http://pkg.jenkins-ci.org/debian/jenkins-ci.org.key | apt-key add -
echo deb http://pkg.jenkins-ci.org/debian binary/ > /etc/apt/sources.list.d/jenkins.list
2) update apt's cache
3) The cache has been updated. Now we can proceed with installing Jenkins
apt-get install jenkins
Once the installation is done you can open Jenkins UI with YOURPUBLICIP:8080
If the jenkins page does not open you can login into the server where you have installed jenkins and check the Jenkins log
tail -f /var/log/jenkins/jenkins.log
sudo service jenkins status
1) Now open the page YOURPUBLICIP:8080
Well, the Jenkins is not secure now. To secure you can go to
Manage Jenkins -> Configure Global Security -> tick Enable Security Under Security Realm -> tick Jenkin’s own user database & Under Authorization -> tick Logged-in users can do anything.
This will take you to the signup page.
2) Since we are using GIT as version control tool and AWS Code Deploy as Deployment tool, we need to install these plugins in Jenkins Manage Plugins option.
to install the plugin
Go to Jenkins Manage Jenkins -> Manage Plugins
In the Available tab search and select GitHub Plugin and AWS CodeDeploy plugin
After selecting click Download and Install after restart
This will download and install both the plugins and restart Jenkins .
2) Its time to create a project in Jenkins and implement CI / CD
Before we go ahead with the further configuration. FYI, My server which contains Jenkins has all the dependencies installed that are required by my app code to build.
>> Create a freestyle project by clicking New Item option
>> Now lets get our git repo url for our project. You can paste your own repo url and paste it in the GitHub Project box
>> In the advanced tab you can mention any specific directory path where you would like jenkins to build the application.
In this case i have kept it to default.
>>Next is you need to fill in the Repository URL with Credentials
Along with the branch in the Branch Specifier
>> In the Build Triggers options you can select Build when a change is pushed to GitHub
What this will do is whenever a new update is pushed to the specified branch in my case */feature/cicd, Jenkins will automatically get triggered, pull the latest updates from the git repo and perform build actions.
The Build option contains build steps, in my case i have selected Execute Shell.
The Command box contains a shell script saved in my jenkins home directory.
This script contains commands to run test case for the application.
If the test case fail then Jenkins would not go further to deploy or won’t perform any further actions and report as Build Failed.
In other case if the test case run successful then you are done Continuous Integrating.
Jenkins will go to next step that is Continuous Deployment.
>> Now for CD, under Add Build Step, select Deploy an application to AWS Code Deploy
In AWS CodeDeploy Application Name : Fill in your AWS CodeDeploy application name that you have configured before.
In AWS CodeDeploy Deployment Group : Fill in your AWS CodeDeploy Deployment Group name that you have configured before.
In AWS CodeDeploy Deployment Config : Paste the Deployment Config you have selected in application configuration. In my case CodeDeployDefault.OneAtATime
In the AWS Regions option : Choose your specific region N.Virginia or Oregon
In the S3 Bucket : Enter your S3 bucket name.
In the S3 Prefix : Enter your directory name under the S3 bucket
Keep everything else to default.
Except you need to give your aws credentials
Fill in you Secret Key and Access Key
SETUP GITHUB AND WEBHOOK
3) To finish setting up the integration and deployment
Go to the GitHub repo, and click Settings. Click the Webhooks & Services tab, and then the Add service drop-down. Select the Jenkins (GitHub plugin) service.
Click Add service. Our project is now ready for its first continuous integration test!
Finally Save your project.
4) This will bring you to a dashboard page where it show you all the list of Projects that are created. Further you can select any Project and Manage, Configure , Build etc.
5) Now its time to Build your project.
So, here on my local machine i commit a code update to my git repo and push it to cicd branch.
This will automatically trigger Jenkins and you will see that it has started to build the project automatically.
If the build is successfull Jenkins will create a zip file of your updated application and push it to AWS S3 and further trigger AWS CodeDeploy application.
You will see a zip file created in S3 folder.
<Original Article written by - Pranav Shah, DevOps Engineer, Cuelogic>
January 22, 2019
How has Azure become so popular in the Enterprise Space? Azure Cloud Services has not… Continue reading...
December 14, 2018
The highlight of tech events this year was certainly AWS re:Invent. The conference was buzzing… Continue reading...
January 4, 2018
Being a CTO and a solution architect for many years I am not at all… Continue reading...
December 4, 2017
Designing or developing any cloud-based application is not a simple straightforward process and always involve… Continue reading...
November 12, 2018
Choosing a web host for your website is a big decision as it is an… Continue reading...
May 9, 2016
“Hybrid cloud computing enables an enterprise to balance isolation, cost and scaling requirements. To implement… Continue reading...