MLOps- git-jenkins integration

This is an assignment given by #linuxworld during MLOps training.

Problem Statement: Automation framework that github and jenkins to look for changes made to dev branch needs to be locally deployed and verified, parallel tested by QA team separately and only after the verification it has to be deployed to Production.

Assumptions: Some of the assumptions are made in this entire solution:

• There are only 2 branches to compare and are available readily
• Git repo has a file like .html which can be easily deployed to apache and verified quickly
• A simple httpd docker is used to deploy this html file
• Verification of the application is to navigate to the html from the specific docker IP:80 and making sure changes are reflecting

Solution Architecture:

Below is the typical branching, versioning and commit lifecycle:

So the problem statement is something like below:

Solution Overview:

In order to meet the desired solution here are the pre-requisites:

• A Linux system with docker-ce installed
• A system where jenkins is installed and has access to docker system
• A system which has access to both the above systems and has browser to open and administer jenkins webui and gthub repo
• All the above systems can still be a one single VM/machiine or different

JOB-1- Dev branch change trigger and verification job:

This is jenkins job which is responsible for automatically identifying the changes on dev branch and deploy the changes locally to a docker container (already running) and test. This simulates a typical developer activities as unit testing. Here are the steps followed to create the jenkins job:

• Create a new job/item in jenkins with freestyle type
• For better housekeeping, Discard build of say 3–5 days (optional)

• In the SCM section provide details of the repo, credentials (https creds or ssh keys as per your convenience) along with branch name as */dev

• Always select option Clean before checkout for better results, Then add poll scm section for every minute

• Then add the Build Activities as shell commands

• Here after the checkout, madhuri.html file is being copied to the docker container’s apache2/htdocs folder and then restart of that container to reflect the changes.Since these activities are for local and developer is performing unit testing, there is no need to send any email
• Once you create and build the job, it will execute only if there are changes in the dev branch and typical output for the Successful job will be like this:

JOB-2-Dev branch change trigger and verification job for QA Team:

• For this case, instead of local and existing docker a new docker container will be created, deployed the changes and let QA test and validate this change and then cleans the docker
• Job steps are almost the same but the changes are in SCM details: as ssh keys as we are supposed to merge the dev to master

• And then the change is on build script:
• Here is the explanation: first lets try deleting any references for index
• checkout and create docker container with the workspace contents mapped/mounted as htdocs
• Then the testing has to be done. For this run the command docker inspect test and you will get the IP address of the container. Using that IP:80/madhuri.html should show the changes made like here in this case its title change from MK7 to MK8

• Once this is tested (time bound activity) the docker is removed Thne the master branch will be checked out and dev will be merged to master along with git push
• Since this involves email notification, post build activity can be email sent along with build logs

If you need more information on the job details and other artifacts, please use Extended Email Plugin

JOB-3- Production deployment job after master branch push:

Once the verification of the dev branch on a fresh system by QA is completed and certified, we need a job to push this to production system. This job is almost same as JOB-1 but changes are:

• In JOB-3 SCM polls the master branch where as JOB-1 polls dev branch
• JOB-3 and JOB-1 docker container names are different
• JOB-3 has Post Build Email notification where JOB-1 doesnt have
• JOB-3 may or may not be automated (as it must be a planned release)

Future advancements to the solution:

This solution is re-usable with minor to major changes as below to suite to many usecases:

• In case of shell script one can write python or boto or even native groovy scripts
• This was example of httpd docker container, similar deployment can also be done for other micorservices on any custom ports
• Verification can also be automated with tools like slenium scripts or groovy or even with simple commands like curl
• Docker images can be built custom and with proper manifest file along with deamon.json with much more secure and standardized parameters
• All the jobs can be nested within a single job or auto trigger by job rather than branches can also be accomplished