One of the lesser talked about benefits of Infrastructure as Code is the ability to use automated testing for verify our configuration before it ever leaves our workstation. Test Kitchen makes this exceptionally easy, and with Docker and kitchen-docker, we can even use Docker containers for our local testing.Using containers for testing is awesome. They are lightweight and fast to create and destroy, but that doesn’t mean they are perfect. One of the most common issues people run into when they start working with Docker containers and Kitchen is the default inability to run services inside of containers. That’s the main issue that we’re going to set out to solve for both CentOS and Ubuntu containers in this short blog post.
Setting Up Kitchen, Docker, and kitchen-docker
Before we get too far, we need to make sure that we have our development environment set up properly. To begin you’ll need to install the following to your workstation:
Once those are installed we’re ready to install kitchen-docker
so that we can use Kitchen and Docker in combination. This package is a Ruby gem, so we’ll want to install it using the gem
command, but we need to make sure that we’re installing it to the group of Chef DK managed gems. To ensure that it is installed in the proper place we’ll use the chef gem
command:
$ chef gem install kitchen-docker
Kitchen comes packaged with the Chef DK so we don’t need to do anything extra to install that. It’s time to create a cookbook and write some tests.
Testing a Wrapper Cookbook
Our example cookbook is going to be a wrapper cookbook around the community nginx cookbook that customizes the service to gzip responses. Since we’re starting from scratch, we also need to create a chef-repo for our cookbooks. Let’s generate the repo and our first cookbook using the chef generate
command:
$ chef generate repo chef-repo...$ cd chef-repochef-repo $ chef generate cookbook cookbooks/custom_nginx...chef-repo $ cd cookbooks/custom_nginx
By default, the chef generate cookbook
command will use a generator that includes a .kitchen.yml file, which will configure how Kitchen will be used in this cookbook. To get started, the only change that we’re going to make is to the driver
, by changing the name
to docker
~/chef-repo/cookbooks/custom_nginx/.kitchen.yml
---driver: name: dockerprovisioner: name: chef_zero # You may wish to disable always updating cookbooks in CI or other testing environments. # For example: # always_update_cookbooks: <%= !ENV['CI'] %> always_update_cookbooks: trueverifier: name: inspecplatforms: - name: ubuntu-16.04 - name: centos-7suites: - name: default run_list: - recipe[custom_nginx::default] verifier: inspec_tests: - test/integration/default attributes:
Note: If your workstation user doesn’t need sudo
to run docker commands, then your driver
section will look like this:
driver: name: docker use_sudo: false
Next, we’re going to customize our default integration test to ensure that the NGINX service is running:~/chef-repo/cookbooks/custom_nginx/test/integration/default/default_test.rb
describe service('nginx') do it { should be_installed } it { should be_running } it { should be_enabled }enddescribe file('/etc/nginx/nginx.conf') do its('content') { should match(%r{gzips+on}) }end
Finally, let’s run our tests for the first time using kitchen test
. This command might take a minute or two, but it’s creating a new node, converging the chef-client, running our tests, and then removing the container:Note: This must be run from within the custom_nginx directory.
custom_nginx $ kitchen test...Profile: tests from {:path=>"/home/user/chef-repo/cookbooks/custom_nginx/test/integration/default"} (tests from{:path=>".home.user.chef-repo.cookbooks.custom_nginx.test.integration.default"})Version: (not specified)Target: ssh://kitchen@localhost:32771 Service nginx ∅ should be installed expected that `Service nginx` is installed ∅ should be running expected that `Service nginx` is running ∅ should be enabled expected that `Service nginx` is enabled File /etc/nginx/nginx.conf ∅ content should match /gzips+on/ expected nil to match /gzips+on/Test Summary: 0 successful, 4 failures, 0 skipped>>>>>> ------Exception------->>>>>> Class: Kitchen::ActionFailed>>>>>> Message: 2 actions failed.>>>>>> Verify failed on instance <default-ubuntu-1604>. Please see .kitchen/logs/default-ubuntu-1604.log f$r more details>>>>>> Verify failed on instance <default-centos-7>. Please see .kitchen/logs/default-centos-7.log for mor$ details>>>>>> ---------------------->>>>>> Please see .kitchen/logs/kitchen.log for more details>>>>>> Also try running `kitchen diagnose --all` for configuration
With failing tests in hand, we’ve completed the “red” step of the “red, green, refactor” approach to Test-Driven Development (TDD). Let’s implement our simple wrapper cookbook that more or less relies on the nginx
cookbook’s “default” recipe.~/chef-repo/cookbooks/custom_nginx/recipes/default.rb
node.default['nginx']['init_style'] = 'systemd'node.default['nginx']['gzip'] = 'on'package "openssl"include_recipe "nginx::default"
We’re installing the openssl
package because we happen to know that NGINX needs it to be present. We also need to specify our dependency in the metadata.rb
for our cookbook and install it locally:~/chef-repo/cookbooks/custom_nginx/metadata.rb
name 'custom_nginx'maintainer 'The Authors'maintainer_email 'you@example.com'license 'All Rights Reserved'description 'Installs/Configures custom_nginx'long_description 'Installs/Configures custom_nginx'version '0.1.0'chef_version '>= 12.14' if respond_to?(:chef_version)depends 'nginx', '~> 8.1.2'
Finally, let’s install the cookbook and rerun our tests:
custom_nginx $ berks install...custom_nginx $ kitchen test...[2018-07-06T20:34:50+00:00] FATAL: Chef::Exceptions::MultipleFailures: Multiple failures occurred: * Chef::Exceptions::Service occurred in chef run: service[nginx] (nginx::package line 50) had an error: Chef::Exceptions::Service: service[nginx]: No custom command for start specified and unable to locate the init.dscript! * Chef::Exceptions::Service occurred in delayed notification: service[nginx] (nginx::package line 50) had an error: Chef::Exceptions::Service: service[nginx]: No custom command for reload specified and unable to locate the init.d script!>>>>>> ------Exception------->>>>>> Class: Kitchen::ActionFailed>>>>>> Message: 2 actions failed.>>>>>> Converge failed on instance <default-ubuntu-1604>. Please see .kitchen/logs/default-ubuntu-1604.logfor more details>>>>>> Converge failed on instance <default-centos-7>. Please see .kitchen/logs/default-centos-7.log for more details>>>>>> ---------------------->>>>>> Please see .kitchen/logs/kitchen.log for more details>>>>>> Also try running `kitchen diagnose --all` for configuration
In both cases, we’re failing the Chef Client run because we can’t start the service. This is a limitation in containers. Since they are designed to run a single process, they don’t run a service manager by default.
Running Services in Containers
All of the code that we have should be enough to get our tests passing, as long as we can get the Chef Client run to complete. To do this, we’re going to modify our .kitchen.yml file a bit more to tweak our containers and Docker. Specifically, we need to ensure that systemd is running, and that the SSH service is started.~/chef-repo/cookbooks/custom_nginx/.kitchen.yml
---driver: name: docker use_sudo: falseprovisioner: name: chef_zero # You may wish to disable always updating cookbooks in CI or other testing environments. # For example: # always_update_cookbooks: <%= !ENV['CI'] %> always_update_cookbooks: trueverifier: name: inspecplatforms: - name: ubuntu-16.04 driver_config: run_command: /bin/systemd privileged: true - name: centos-7 driver_config: run_command: /usr/sbin/init privileged: true provision_command: - systemctl enable sshd.servicesuites: - name: default run_list: - recipe[custom_nginx::default] verifier: inspec_tests: - test/integration/default attributes:
By modifying the driver_config
for each of our platform
items, we’re able to start systemd before we run anything else. Let’s run our tests one last time:
custom_nginx $ kitchen test...Profile: tests from {:path=>"/home/user/chef-repo/cookbooks/custom_nginx/test/integration/default"} (t[52/13089]{:path=>".home.user.chef-repo.cookbooks.custom_nginx.test.integration.default"})Version: (not specified)Target: ssh://kitchen@localhost:32787 Service nginx ✔ should be installed ✔ should be running ✔ should be enabled File /etc/nginx/nginx.conf ✔ content should match /gzips+on/Test Summary: 4 successful, 0 failures, 0 skipped...custon_nginx $
Unfortunately, when the test run finishes, we see a lot of output from the teardown. We need to scroll up a little to see that the tests passed for CentOS. But if we scroll much further up through the output, we can see that the tests also passed for the Ubuntu container. If there had been an error in either of the containers, then the error would be front and center at the bottom of the output, with the command returning a non-zero
exit status.Now you know how to get kitchen-docker
to use containers successfully when you’re testing cookbooks that utilize the service
resource. If you’re using an operating system other than Ubuntu or CentOS you’ll want to follow a similar approach using whatever means the operating system needs to start its service manager.Chef Resources:- Chef Basic Fluency Badge– Chef Local Cookbook Development Badge