MLOps end-to-end deployment of Covid_X-ray prediction

About the Core project:

With available small set of chest X-ray images of Covid patients and other normal chest X-ray images, predicting whether patient is Covid +ve or -ve. (Using binary classification)

Sample X-ray images of both classes

For this, I am using Keras CNN model. But twist here is that, tuning model (say by adding more convolution layer or by tweaking epochs etc.) is attempted via automation.

This approach is carried out for end-to-end machine learning automation deployment by using below MLOps architecture, with the integration of :

GitHub +Jenkins (Build pipeline) +Docker

Pre-Requisites:

1. Custom created relevant docker image. (Docker image creation for Keras refer: https://medium.com/@magicwordsmadhu/docker-image-creation-60448b3b546d )

2. Configured Jenkins for build pipeline jobs. Refer: https://www.jenkins.io/pipeline/getting-started-pipelines/

3. Configured Jenkins to git ssh credentials. Refer: https://mohitgoyal.co/2017/02/27/configuring-ssh-authentication-between-github-and-jenkins/

Note: Build steps for all Jenkin jobs are written in shell script. Other options are window’s batch script, ant, maven, gradle or groovy

Problem Statement:

MLOps -Architecture

Build pipeline job description:

Job 1. Download code from git

Job 2 . Check the file for cnn and pull the docker

Then start training model inside docker.

Job 3. Training the cnn model and predicting the accuracy.

Below is the snippet of the job Build Execution section (using shell):

pwd
#Get the accuracy value
ACC”Accuracy is $ACC”
#Minimu=`docker exec -t c1 cat accuracy.txt`
echo m accuracy required is 0.95
MIN=0.95
#Check if the accuracy is less than .95 add another convolution layer
if [ 1 -eq “$(echo “${ACC} < ${MIN}” | tr -d $’\r’ | bc)” ]; then echo “Accuracy is less than $MIN”; echo “Adding convolution layer”; sed -i “/model.add(Flatten.*/i model.add(Conv2D(64,(3,3),activation=’relu’ )) “ covid_mk1.py; sed -i ‘/model.add(Flatten.*/i model.add(MaxPooling2D(pool_size=(2,2))) ‘ covid_mk1.py; sed -i ‘/model.add(Flatten.*/i model.add(Dropout(0.25)) ‘ covid_mk1.py; git commit covid_mk1.py -m “Added convulution layer”; git push origin HEAD:master ; else echo “Best Created Model..” ; fi ;

Job 4. Tweaking the cnn model based on accuracy

This results in pipeline execution again until the desired accuracy is met.

Starting of Build pipeline

All jobs success in build pipeline

Initial git repo view before pipeline run

Observe above flatten layer. No layers are added initially

During model running …

Jenkins job console output

model training

Starting accuracy of model

(Here we have set desired accuracy greater than or equals to 95%)

Final view:

Once after tweaking model, pushing back to git automatically triggers pipeline, cycle repeats

Auto git push from the job showing the additional layer

Obtained desired accuracy of model

Model preparation with dataset source:

Her is the link to the detailed documentation on how to prepare the dataset for model: https://github.com/MadhuriKonnur/Covid-X-ray

Future enhancements:

1. Job in pipeline can have different options to choose between models like CNN, RNN etc.

2. Build push notification and component monitoring like docker jobs can be added

3. Front end application can be built on saved model.

4. Typical jenkinsfile based Build Pipeline (groovy or python or go etc)

5. Jenkins within docker based pipeline

Please feel free to review and let me know any inputs or feedback.