Jenkins Pipelines are basically an automation feature which offer versatility when one is testing, building, and deploying code. In this hands-on lab we will be building a pipeline which will utilize multiple Jenkins build agents for carrying out different parts of a Jenkins job.
Learning Objectives
Successfully complete this lab by achieving the following learning objectives:
- Jenkins UI (Verify That Jenkins Worker Nodes Are Integrated)
Once you access the landing page for Jenkins (http://<Public-IP-of-Jenkins>:8080), click Manage Jenkins on the left side of the screen. In the new screen that loads, look for and click on Manage Nodes and Clouds. On this new page that loads you should see three systems:
master
JenkinsWorker1
JenkinsWorker2
- Click the bell icon.
- Apply the Migration.
- Click the bell icon.
- Choose Manage Jenkins.
- Choose Go to plugin manager.
- Select all, Download now and install after restart.
- After all the items on the list are marked with an action, Checkmark "Restart Jenkins when installation is complete and no jobs are running."
- Reconnect to Jenkins.
- You’ll now see both Worker nodes as Idle.
Note that there should be no red x symbol on any of the systems on this page, indicating that connectivity is good between them and the Jenkins master.
master
is the Jenkins instance hosting your Jenkins UI.JenkinsWorker1
has been assigned the label docker1.JenkinsWorker2
has been assigned label docker2.We’ll be using these labels later in the tasks to assign jobs to these Jenkins agents.
We will be creating two separate pipelines which will work in unison to complete a task. Both of these pipelines will run on separate Jenkins workers/agent nodes, which already have been provisioned and integrated with the Jenkins
master
instance you’re working on.Before you start creating pipelines and using these workers in those pipelines, go ahead and verify their connectivity.
- Configure Pipeline # 1
From the Manage Nodes and Clouds page, click Back to Dashboard on the left options pane. Once you’re back on the dashboard, click on New item to create a new pipeline. Click on Pipeline, and in the input box give your first pipeline job name pipeline1. Make sure that you give it this name.
After selecting the type of job, Pipeline, and giving it a name, click on OK to proceed with configuring the job.
On the new page which loads for configuring the pipeline job, scroll down to the Pipeline section and ensure that Pipeline Script is selected in the Definition drop-down. An input area should be visible. Copy the following body of text (which is basically in Groovy DSL format, a format understood by Jenkins) into the input area:
pipeline{ agent { docker { image 'python:alpine' args '-u root' label 'docker1' } } stages{ stage('prep'){ steps{ sh 'apk add git' } } stage('fetch'){ steps{ sh 'git clone https://github.com/linuxacademy/content-pipelines-cje-labs.git' } } stage('add'){ steps{ sh 'pip install -r content-pipelines-cje-labs/lab3_lab4/dog_pics_downloader/requirements.txt' } } stage('execute'){ steps{ sh 'python content-pipelines-cje-labs/lab3_lab4/dog_pics_downloader/dog_pic_get_class.py' } } } post{ always{ echo "execution complete" } success{ archiveArtifacts artifacts: '*.jpg' } cleanup{ sh 'rm -rf content-pipelines-cje-labs' } } }
Leave everything else at default settings and click Save.
Note that in the agent block for the pipeline, we have instructed the container to be run on the
JenkinsWorker1
worker/agent with label docker1.- Configure Pipeline # 2
Once you’ve created the first pipeline, click on the Back to Dashboard button and head back to the New Item page for creating the second pipeline. Select Pipeline and give it an appropriate name, such as pipeline2.
On this pipeline’s configuration page, go to the General section. Check and enable plugin Permission to Copy Artifact. This will show a new input form area where you’ll enter the name of the pipeline from which you want to copy saved artifacts. In our case, the name of the other pipeline is
pipeline1
. Make sure to type the name of the pipeline out fully, or click on the drop-down autofill option.Under the Build Triggers section, check and enable the Build after other projects are built option. This should provide more input options. Fill them out as follows:
- For Projects to watch, type in pipeline1
- Click on the radio button Trigger only if build is stable. This ensures that this second pipeline will trigger on the successful/stable build of
pipeline1
.
Under the Pipeline section ensure that Pipeline Script is selected as a Definition, and paste the following snippet into the input area:
pipeline{ agent { docker{ image 'centos:7' args '-u root' label 'docker2' } } stages{ stage('dependencies'){ steps{ sh 'yum -y install python3 python3-pip zlib-devel gcc git deltarpm ' } } stage('copyart'){ steps{ copyArtifacts(projectName: 'pipeline1', flatten: true) } } stage('fetch'){ steps{ sh 'git clone https://github.com/linuxacademy/content-pipelines-cje-labs.git' } } stage('install'){ steps{ sh 'pip3 install -r content-pipelines-cje-labs/lab3_lab4/image_watermarker/requirements.txt' } } stage('exec'){ steps{ sh 'python3 content-pipelines-cje-labs/lab3_lab4/image_watermarker/watermark.py' } } } post{ success{ archiveArtifacts artifacts: '*watermarked.jpg' } cleanup{ sh 'rm -rf content-pipelines-cje-labs *.jpg' } } }
Leave everything else at the default settings and click on Save. Note that in the agent block for the pipeline, we have instructed the container to be run on the
JenkinsWorker2
worker/agent with label docker2.- Execute the First Pipeline Build (Which Will Kick off the Second Pipeline)
Head back to the Dashboard, which shows all jobs, by clicking on the Back to Dashboard button (if you’re not already on this page).
In the
pipeline1
row, over to the right, click on the button with a green arrow to kick off the job. Once this job completes, the second pipeline job will be executed automatically. The progress of both these jobs will be visible on the Dashboard page under Build Executor Status widget to the right side of the same page. This will show all the Jenkins machines and any job steps that are running on them.If everything runs successfully, the success indicator ball icon for both jobs (which are right next to the job names) will turn blue.
Note: You may need to refresh the page to see the ball indicators change color.
- Verify That Both Pipelines Ran Successfully (Check Produced Artifacts)
On the Dashboard page, you can click on either one of the pipelines to go into the details of that job and check the artifacts produced by the job, as well as go down into specific build number and check console output for that build.
pipeline1
will produce somejpg
files and archive them.pipeline2
(or whatever you named the second pipeline) will copy over the artifacts orjpg
files produced bypipeline1
, and will generate newjpg
files that are watermarked with information. If you want to view ajpg
file, click on the file name to open it in browser. Do not click on view, as it will show the file’s raw data.