Testing Chef Cookbooks with Kitchen and Docker

One of the lesser talked about benefits of Infrastructure as Code is the ability to use automated testing for verify our configuration before it ever leaves our workstation. Test Kitchen makes this exceptionally easy, and with Docker and kitchen-docker, we can even use Docker containers for our local testing.Using containers for testing is awesome. They are lightweight and fast to create and destroy, but that doesn’t mean they are perfect. One of the most common issues people run into when they start working with Docker containers and Kitchen is the default inability to run services inside of containers. That’s the main issue that we’re going to set out to solve for both CentOS and Ubuntu containers in this short blog post.

Setting Up Kitchen, Docker, and kitchen-docker

Before we get too far, we need to make sure that we have our development environment set up properly. To begin you’ll need to install the following to your workstation:

• Docker
• Chef DK

Once those are installed we’re ready to install kitchen-docker so that we can use Kitchen and Docker in combination. This package is a Ruby gem, so we’ll want to install it using the gem command, but we need to make sure that we’re installing it to the group of Chef DK managed gems. To ensure that it is installed in the proper place we’ll use the chef gem command:

$ chef gem install kitchen-docker

Kitchen comes packaged with the Chef DK so we don’t need to do anything extra to install that. It’s time to create a cookbook and write some tests.

Testing a Wrapper Cookbook

Our example cookbook is going to be a wrapper cookbook around the community nginx cookbook that customizes the service to gzip responses. Since we’re starting from scratch, we also need to create a chef-repo for our cookbooks. Let’s generate the repo and our first cookbook using the chef generate command:

$ chef generate repo chef-repo...$ cd chef-repochef-repo $ chef generate cookbook cookbooks/custom_nginx...chef-repo $ cd cookbooks/custom_nginx

By default, the chef generate cookbook command will use a generator that includes a .kitchen.yml file, which will configure how Kitchen will be used in this cookbook. To get started, the only change that we’re going to make is to the driver, by changing the name to docker~/chef-repo/cookbooks/custom_nginx/.kitchen.yml

---driver:  name: dockerprovisioner:  name: chef_zero  # You may wish to disable always updating cookbooks in CI or other testing environments.  # For example:  #   always_update_cookbooks: <%= !ENV['CI'] %>  always_update_cookbooks: trueverifier:  name: inspecplatforms:  - name: ubuntu-16.04  - name: centos-7suites:  - name: default    run_list:      - recipe[custom_nginx::default]    verifier:      inspec_tests:        - test/integration/default    attributes:

Note: If your workstation user doesn’t need sudo to run docker commands, then your driver section will look like this:

driver:  name: docker  use_sudo: false

Next, we’re going to customize our default integration test to ensure that the NGINX service is running:~/chef-repo/cookbooks/custom_nginx/test/integration/default/default_test.rb

describe service('nginx') do  it { should be_installed }  it { should be_running }  it { should be_enabled }enddescribe file('/etc/nginx/nginx.conf') do  its('content') { should match(%r{gzips+on}) }end

Finally, let’s run our tests for the first time using kitchen test. This command might take a minute or two, but it’s creating a new node, converging the chef-client, running our tests, and then removing the container:Note: This must be run from within the custom_nginx directory.

custom_nginx $ kitchen test...Profile: tests from {:path=>"/home/user/chef-repo/cookbooks/custom_nginx/test/integration/default"} (tests from{:path=>".home.user.chef-repo.cookbooks.custom_nginx.test.integration.default"})Version: (not specified)Target:  ssh://kitchen@localhost:32771  Service nginx     ∅  should be installed     expected that `Service nginx` is installed     ∅  should be running     expected that `Service nginx` is running     ∅  should be enabled     expected that `Service nginx` is enabled  File /etc/nginx/nginx.conf     ∅  content should match /gzips+on/     expected nil to match /gzips+on/Test Summary: 0 successful, 4 failures, 0 skipped>>>>>> ------Exception------->>>>>> Class: Kitchen::ActionFailed>>>>>> Message: 2 actions failed.>>>>>>     Verify failed on instance <default-ubuntu-1604>.  Please see .kitchen/logs/default-ubuntu-1604.log f$r more details>>>>>>     Verify failed on instance <default-centos-7>.  Please see .kitchen/logs/default-centos-7.log for mor$ details>>>>>> ---------------------->>>>>> Please see .kitchen/logs/kitchen.log for more details>>>>>> Also try running `kitchen diagnose --all` for configuration

With failing tests in hand, we’ve completed the “red” step of the “red, green, refactor” approach to Test-Driven Development (TDD). Let’s implement our simple wrapper cookbook that more or less relies on the nginx cookbook’s “default” recipe.~/chef-repo/cookbooks/custom_nginx/recipes/default.rb

node.default['nginx']['init_style'] = 'systemd'node.default['nginx']['gzip'] = 'on'package "openssl"include_recipe "nginx::default"

We’re installing the openssl package because we happen to know that NGINX needs it to be present. We also need to specify our dependency in the metadata.rb for our cookbook and install it locally:~/chef-repo/cookbooks/custom_nginx/metadata.rb

name 'custom_nginx'maintainer 'The Authors'maintainer_email 'you@example.com'license 'All Rights Reserved'description 'Installs/Configures custom_nginx'long_description 'Installs/Configures custom_nginx'version '0.1.0'chef_version '>= 12.14' if respond_to?(:chef_version)depends 'nginx', '~> 8.1.2'

Finally, let’s install the cookbook and rerun our tests:

custom_nginx $ berks install...custom_nginx $ kitchen test...[2018-07-06T20:34:50+00:00] FATAL: Chef::Exceptions::MultipleFailures: Multiple failures occurred:       * Chef::Exceptions::Service occurred in chef run: service[nginx] (nginx::package line 50) had an error: Chef::Exceptions::Service: service[nginx]: No custom command for start specified and unable to locate the init.dscript!       * Chef::Exceptions::Service occurred in delayed notification: service[nginx] (nginx::package line 50) had an error: Chef::Exceptions::Service: service[nginx]: No custom command for reload specified and unable to locate the init.d script!>>>>>> ------Exception------->>>>>> Class: Kitchen::ActionFailed>>>>>> Message: 2 actions failed.>>>>>>     Converge failed on instance <default-ubuntu-1604>.  Please see .kitchen/logs/default-ubuntu-1604.logfor more details>>>>>>     Converge failed on instance <default-centos-7>.  Please see .kitchen/logs/default-centos-7.log for more details>>>>>> ---------------------->>>>>> Please see .kitchen/logs/kitchen.log for more details>>>>>> Also try running `kitchen diagnose --all` for configuration

In both cases, we’re failing the Chef Client run because we can’t start the service. This is a limitation in containers. Since they are designed to run a single process, they don’t run a service manager by default.

Running Services in Containers

All of the code that we have should be enough to get our tests passing, as long as we can get the Chef Client run to complete. To do this, we’re going to modify our .kitchen.yml file a bit more to tweak our containers and Docker. Specifically, we need to ensure that systemd is running, and that the SSH service is started.~/chef-repo/cookbooks/custom_nginx/.kitchen.yml

---driver:  name: docker  use_sudo: falseprovisioner:  name: chef_zero  # You may wish to disable always updating cookbooks in CI or other testing environments.  # For example:  #   always_update_cookbooks: <%= !ENV['CI'] %>  always_update_cookbooks: trueverifier:  name: inspecplatforms:  - name: ubuntu-16.04    driver_config:      run_command: /bin/systemd      privileged: true  - name: centos-7    driver_config:      run_command: /usr/sbin/init      privileged: true      provision_command:        - systemctl enable sshd.servicesuites:  - name: default    run_list:      - recipe[custom_nginx::default]    verifier:      inspec_tests:        - test/integration/default    attributes:

By modifying the driver_config for each of our platform items, we’re able to start systemd before we run anything else. Let’s run our tests one last time:

custom_nginx $ kitchen test...Profile: tests from {:path=>"/home/user/chef-repo/cookbooks/custom_nginx/test/integration/default"} (t[52/13089]{:path=>".home.user.chef-repo.cookbooks.custom_nginx.test.integration.default"})Version: (not specified)Target:  ssh://kitchen@localhost:32787  Service nginx     ✔  should be installed     ✔  should be running     ✔  should be enabled  File /etc/nginx/nginx.conf     ✔  content should match /gzips+on/Test Summary: 4 successful, 0 failures, 0 skipped...custon_nginx $

Unfortunately, when the test run finishes, we see a lot of output from the teardown. We need to scroll up a little to see that the tests passed for CentOS. But if we scroll much further up through the output, we can see that the tests also passed for the Ubuntu container. If there had been an error in either of the containers, then the error would be front and center at the bottom of the output, with the command returning a non-zero exit status.Now you know how to get kitchen-docker to use containers successfully when you’re testing cookbooks that utilize the service resource. If you’re using an operating system other than Ubuntu or CentOS you’ll want to follow a similar approach using whatever means the operating system needs to start its service manager.Chef Resources:- Chef Basic Fluency Badge– Chef Local Cookbook Development Badge