CI/CD: How to switch from Manual to Automated development

By Nivedita Sood

In DevOps

18 Jun 2020

6 min read

When speaking of manual and automatic development, the importance of CI/CD is no longer in question. Speed and reliability are the two main benefits of CI/CD.

In this blog I am sharing what made me do CI/CD and all the ins and outs one needs to implement CI/CD pipeline in their projects. The post will also cover how you can automate the deployment to AWS cloud using Gitlab CI/CD service.

What is CI/CD?

CI/CD is short for Continuous Integration/ Continuous Delivery/Continuous Deployment. It means shipping small and frequent changes with
automated builds and tests. CI/CD removes manual human interactions where possible – automating everything and gives greater visibility into the process.

Why did I use CI/CD?

There are two important things in CI/CD. First is to build fast, reliable and frequent integrations which is CI. Second is to automate deployment, make it a push-button operation, and make it easy to test code in production-like environments which is CD.

In one of the projects I was working on manual deployment posed a problem. Shipping that code was a painful process. Simple bug fixation took a laborious 3-4 hrs route to production and an excessive coordination was needed during production deployments.

Now, who wants to take care of manually deploying every single feature? What if there are three new features tested every day? I felt the pinch. I was keen to move to automated deployment. I brainstormed with other stakeholders of the project for the automation possibilities. And so we began..

In the beginning, it was a big teething problem, as I was new on the project and wasn’t fully aware of the production ec2 setup of the application I was about to automate. But the benefits of CI/CD kept me motivated throughout the project.

The awesome Devops team @Shaadi was already doing CI/CD for other projects, so I got great help from them to kick off my project.

How I switched to CI/CD?

You can’t jump headlong into a CI/CD project without knowing exactly where your starting point is. Here are the steps I took to complete my journey to CI and CD.

First, I reviewed everything about the project and all the dependencies. I audited all the moving parts from branch commit to production environment deployments. This helped me clarify exactly what I needed to deal with and what change I could bring.
I read the articles to create a docker. While building docker I kept only one thing in mind that it should reciprocate our current ec2 server where the existing application runs. Again this ensured that nothing will break when we go live with the CI/CD pipeline.
I then explored Gitlab CI/CD service to build and test the application on code push.
Read many articles on how Gitlab CI/CD talks with AWS cloud.
Lastly, I kept my project scope to build automated deployment to production without manual push. It’s good to not include everything in the automated build in the first iteration of the project. I just built a system without automated tests.

Let’s now demystify CI/CD pipeline setup for container deployments on AWS.

CICD pipeline – Build, Ship, Run , Any App, Anywhere

The step by step guide below will walk you through all the phases talked above and ease your CI/CD pipeline creation.

Prerequisites:

Install docker
Get the Gitlab Registration Token
ECR repo on AWS
AWS IAM user for Gitlab
AWS EC2 instance

This may still sound like a big hill to climb. However, the results will be worth it.

How to Dockerise your App?

Docker helps developers build lightweight and portable software containers that simplify application development, testing, and deployment. It creates a portable image.

In dockerising any app, first you build docker image and then deploy the image as docker container.

To dockerize your app you need to create a Dockerfile. It’s a text file which includes the instructions to build a Docker image.

A sample file for my app is shown below:

Here I have instructed docker which base image to use for the container, install Python 3.6, set code working directory, install all other dependencies of my app and then finally run the app when container starts.

How to build CICD pipeline in Gitlab?

You need to install GitLab Runner on server.

How to setup Gitlab runner?

To install Gitlab runner: follow this link
Start as service: sudo /usr/local/bin/gitlab-runner start
Register the Runner. follow this link.
You can now see the Registered Runner in your Gitlab account:
Project/Group -> Settings -> CI/CD -> Runners

How to setup AWS and Gitlab communication?

Create new AWS IAM user for Gitlab
Setup environment variables with the AWS keys
- you need two variables: “SSH_PRIVATE_KEY” and “DEPLOY_SERVER_IP”. You can create variables from “settings > CI/CD Pipelines”

How to push docker image in the AWS ECR Registry and deploy image on your EC2 servers?

To enable GitLab CI/CD feature you have to place .gitlab-ci.yml file in the Root of your code Repository. On any push to your repository, GitLab will look for the .gitlab-ci.yml file and start jobs on Runners according to the contents of the file, for that commit. In the file we can define deployment stages like test, release, and deploy.

Here is the sample .gitlab-ci.yml file to push to AWS ECR Registry and Deploy code on the AWS instance:

image: docker:latest

services:
- docker:dind

variables:
  REPOSITORY_URL: 515997684604.dkr.ecr.us-east-1.amazonaws.com/test

before_script:
  - apk add --no-cache curl jq python3 py3-pip
  - pip3 install awscli

stages:
  - build
  - production

build:
  stage: build
  script:
    - $(aws ecr get-login --no-include-email --region us-east-1)
    - docker build -t $REPOSITORY_URL .
    - docker push $REPOSITORY_URL

production:
  stage: production
  before_script:
    - mkdir -p ~/.ssh
    - echo "$SSH_PRIVATE_KEY" | tr -d '\r' > ~/.ssh/id_rsa
    - chmod 600 ~/.ssh/id_rsa
  script:
    - ssh -o StrictHostKeyChecking=no -i ~/.ssh/id_rsa ubuntu@$DEPLOY_SERVER_IP docker pull $REPOSITORY_URL:latest
    - ssh -o StrictHostKeyChecking=no -i ~/.ssh/id_rsa ubuntu@$DEPLOY_SERVER_IP docker run -t $REPOSITORY_URL:latest

That’s it!! Now every time a code push is done into the master branch, it would be deployed to all the servers. You can view your complete CI/CD pipeline in Gitlab.