Advanced Jenkins Pipelines

Jenkins Pipelines are basically an automation feature which offer versatility when one is testing, building, and deploying code. In this hands-on lab we will be building a pipeline which will utilize multiple Jenkins build agents for carrying out different parts of a Jenkins job.

Learning Objectives

Successfully complete this lab by achieving the following learning objectives:

Jenkins UI (Verify That Jenkins Worker Nodes Are Integrated)

Once you access the landing page for Jenkins (http://<Public-IP-of-Jenkins>:8080), click Manage Jenkins on the left side of the screen. In the new screen that loads, look for and click on Manage Nodes and Clouds. On this new page that loads you should see three systems:

master
JenkinsWorker1
JenkinsWorker2

Click the bell icon.
Apply the Migration.
Click the bell icon.
Choose Manage Jenkins.
Choose Go to plugin manager.
Select all, Download now and install after restart.
After all the items on the list are marked with an action, Checkmark "Restart Jenkins when installation is complete and no jobs are running."
Reconnect to Jenkins.
You’ll now see both Worker nodes as Idle.

Note that there should be no red x symbol on any of the systems on this page, indicating that connectivity is good between them and the Jenkins master.

master is the Jenkins instance hosting your Jenkins UI. JenkinsWorker1 has been assigned the label docker1. JenkinsWorker2 has been assigned label docker2.

We’ll be using these labels later in the tasks to assign jobs to these Jenkins agents.

We will be creating two separate pipelines which will work in unison to complete a task. Both of these pipelines will run on separate Jenkins workers/agent nodes, which already have been provisioned and integrated with the Jenkins master instance you’re working on.

Before you start creating pipelines and using these workers in those pipelines, go ahead and verify their connectivity.

Configure Pipeline # 1

From the Manage Nodes and Clouds page, click Back to Dashboard on the left options pane. Once you’re back on the dashboard, click on New item to create a new pipeline. Click on Pipeline, and in the input box give your first pipeline job name pipeline1. Make sure that you give it this name.

After selecting the type of job, Pipeline, and giving it a name, click on OK to proceed with configuring the job.

On the new page which loads for configuring the pipeline job, scroll down to the Pipeline section and ensure that Pipeline Script is selected in the Definition drop-down. An input area should be visible. Copy the following body of text (which is basically in Groovy DSL format, a format understood by Jenkins) into the input area:

pipeline{
    agent {
        docker {
            image 'python:alpine'
            args '-u root'
            label 'docker1'
        }
    }
    stages{
        stage('prep'){
            steps{
                sh 'apk add git'
            }
        }
        stage('fetch'){
            steps{
            sh 'git clone https://github.com/linuxacademy/content-pipelines-cje-labs.git'                
            }
        }
        stage('add'){
            steps{
                sh 'pip install -r content-pipelines-cje-labs/lab3_lab4/dog_pics_downloader/requirements.txt'
            }
        }
        stage('execute'){
            steps{
                sh 'python content-pipelines-cje-labs/lab3_lab4/dog_pics_downloader/dog_pic_get_class.py'
            }
        }
    }
    post{
        always{
            echo "execution complete"
        }
        success{
            archiveArtifacts artifacts: '*.jpg'
        }
        cleanup{
            sh 'rm -rf content-pipelines-cje-labs'
        }
    }
}

Leave everything else at default settings and click Save.

Note that in the agent block for the pipeline, we have instructed the container to be run on the JenkinsWorker1 worker/agent with label docker1.

Configure Pipeline # 2

Once you’ve created the first pipeline, click on the Back to Dashboard button and head back to the New Item page for creating the second pipeline. Select Pipeline and give it an appropriate name, such as pipeline2.

On this pipeline’s configuration page, go to the General section. Check and enable plugin Permission to Copy Artifact. This will show a new input form area where you’ll enter the name of the pipeline from which you want to copy saved artifacts. In our case, the name of the other pipeline is pipeline1. Make sure to type the name of the pipeline out fully, or click on the drop-down autofill option.

Under the Build Triggers section, check and enable the Build after other projects are built option. This should provide more input options. Fill them out as follows:

For Projects to watch, type in pipeline1
Click on the radio button Trigger only if build is stable. This ensures that this second pipeline will trigger on the successful/stable build of pipeline1.

Under the Pipeline section ensure that Pipeline Script is selected as a Definition, and paste the following snippet into the input area:

pipeline{
    agent {
        docker{ image 'centos:7'
                args '-u root'
                label 'docker2'
        }
    }

    stages{
        stage('dependencies'){
            steps{
                sh 'yum -y install python3 python3-pip zlib-devel gcc git deltarpm '
            }
    }
        stage('copyart'){
            steps{
                copyArtifacts(projectName: 'pipeline1', flatten: true)
            }
        }
        stage('fetch'){
            steps{
                sh 'git clone https://github.com/linuxacademy/content-pipelines-cje-labs.git'
            }
        }
        stage('install'){
            steps{
                sh 'pip3 install -r content-pipelines-cje-labs/lab3_lab4/image_watermarker/requirements.txt'
            }
        }
        stage('exec'){
            steps{
                sh 'python3 content-pipelines-cje-labs/lab3_lab4/image_watermarker/watermark.py'
            }
        }
    }
    post{
        success{
            archiveArtifacts artifacts: '*watermarked.jpg'
        }
        cleanup{
            sh 'rm -rf content-pipelines-cje-labs *.jpg'
        }
    }
}

Leave everything else at the default settings and click on Save. Note that in the agent block for the pipeline, we have instructed the container to be run on the JenkinsWorker2 worker/agent with label docker2.

Execute the First Pipeline Build (Which Will Kick off the Second Pipeline)

Head back to the Dashboard, which shows all jobs, by clicking on the Back to Dashboard button (if you’re not already on this page).

In the pipeline1 row, over to the right, click on the button with a green arrow to kick off the job. Once this job completes, the second pipeline job will be executed automatically. The progress of both these jobs will be visible on the Dashboard page under Build Executor Status widget to the right side of the same page. This will show all the Jenkins machines and any job steps that are running on them.

If everything runs successfully, the success indicator ball icon for both jobs (which are right next to the job names) will turn blue.

Note: You may need to refresh the page to see the ball indicators change color.

Verify That Both Pipelines Ran Successfully (Check Produced Artifacts)

On the Dashboard page, you can click on either one of the pipelines to go into the details of that job and check the artifacts produced by the job, as well as go down into specific build number and check console output for that build.

pipeline1 will produce some jpg files and archive them. pipeline2 (or whatever you named the second pipeline) will copy over the artifacts or jpg files produced by pipeline1, and will generate new jpg files that are watermarked with information. If you want to view a jpg file, click on the file name to open it in browser. Do not click on view, as it will show the file’s raw data.

Additional Resources

You work for a SaaS company that has been using Jenkins Pipelines for a while now. Recently, it was noted that the Jenkins master was under too much load, as all jobs were being run on it.

Your team lead decided to integrate two Jenkins workers with the Jenkins master, to assist in distributing jobs to be executed on worker nodes. The two new Jenkins workers (JenkinsWorker1 labeled docker1, and JenkinsWorker2 labeled docker2) have been finally setup.

Your Team Lead has asked you to break down a single job which used to spin up multiple containers to test parts of your application.

The first part of your application to be tested fetches some JPG files, and the second part watermarks them. All artifacts must be saved.

You have a rough idea about how to go about making the build more distributed.

First Pipeline

Create a new Jenkins Pipeline named pipeline1 which will run on JenkinsWorker1 (labeled docker1 and to be run with the args '-u root' option). This pipeline will run a containerized job using the python:alpine image to pull down code, set up the Python environment and execute the code, then archive any JPG files that were downloaded. Steps would roughly require the following commands:

Install git inside container:
```
apk add git
```

Clone source code:

git clone https://github.com/linuxacademy/content-pipelines-cje-labs.git

Install modules required by code:

pip install -r content-pipelines-cje-labs/lab3_lab4/dog_pics_downloader/requirements.txt

Execute the code to test it:

python content-pipelines-cje-labs/lab3_lab4/dog_pics_downloader/dog_pic_get_class.py

Archive any files with the .jpg extension. Optionally, for bonus marks, also include success and cleanup (to delete folder content-pipelines-cje-labs) steps in the post block of the pipeline.

Second Pipeline

Create a new Jenkins Pipeline named pipeline2 which will run on JenkinsWorker2 (labeled docker2 and to be run with the args '-u root' option). This one will run a containerized job using the centos:7 image to pull down code, set up the Python environment and execute the code, then archive any watermarked jpg files which are produced that end with the string watermarked.jpg. Steps would roughly require the following commands:

Install required dependencies:

yum -y install python3 python3-pip zlib-devel gcc git

Copy over artifacts produced from the pipeline using the following step command:
```
copyArtifacts(projectName: 'pipeline1', flatten: true)
```

Fetch source code:

git clone https://github.com/linuxacademy/content-pipelines-cje-labs.git

Install module required by code:

pip3 install -r content-pipelines-cje-labs/lab3_lab4/image_watermarker/requirements.txt

Execute code to test it:

python3 content-pipelines-cje-labs/lab3_lab4/image_watermarker/watermark.py

Archive, in post block, any files ending in watermarked.jpg. Optionally, also run a cleanup step in post block to delete any files ending in jpg as well the directory content-pipelines-cje-labs. Ensure the second pipeline uses the Copy Artifact plugin option (checked during Pipeline configuration) to copy artifacts over from pipeline1, and have Build Trigger enabled to be executed whenever pipeline1 job executes successfully (i.e. a stable build). 2nd Pipeline must execute automatically. Verify success by checking that archives and are jpg files. You have the independence to decide how many stages to use for executing the steps in both pipelines.

What are Hands-on Labs

Hands-on Labs are real environments created by industry experts to help you learn. These environments help you gain knowledge and experience, practice without compromising your system, test without risk, destroy without fear, and let you learn from your mistakes. Hands-on Labs: practice your skills before delivering in the real world.