Dokken (Docker)
kitchen-dokken is a Test Kitchen plugin for Docker that uses specially created Linux distribution Docker images and Chef Infra Docker images. Utilizing the Docker or Podman container engines instead of hypervisors or cloud providers allows for Chef Infra cookbook testing that can run on a local workstation or in CI pipelines without modification or extra costs.
Unlike all other Test Kitchen drivers, kitchen-dokken handles all the tasks of the driver, transport, and provisioner itself. This approach focuses purely on Chef Infra cookbook testing to provide ultra-fast testing times. Docker containers have a fast creation and start time, and kitchen-dokken uses the official Chef Infra Client containers instead of spending the time to download and install the Chef Infra Client packages. These design decisions result in tests that run in seconds instead of minutes and don’t require high bandwidth Internet connections.
Driver vs. Provisioner vs. Transport
Since kitchen-dokken combines driver, transport, and provisioner functionality into a single plugin, its configuration differs from other Test Kitchen drivers. kitchen-dokken includes specific configuration options that can be set in the driver, provisioner, or transport sections or in the individual platforms / suites. At a bare minimum to use kitchen-dokken you must specify dokken as the driver, provisioner, and transport sections of your kitchen.yml
file:
---
driver:
name: dokken
provisioner:
name: dokken
transport:
name: dokken
Driver Configuration Options
api_retries
The api_retries
configuration option specifies the number of times Test Kitchen will retry communicating with the Docker daemon before exiting with a failure. This configuration option defaults to 20
.
driver:
name: dokken
api_retries: 30
binds
The binds
configuration option allows you bind mount a local path into your Test Kitchen containers.
driver:
name: dokken
binds:
- /some/local/path:/some/container/path
cap_add
The cap_add
configuration option allows you to specify additional system capabilities to add to the container. See the Docker Runtime Privilege and Linux Capabilities Documentation for a complete list of capabilities.
driver:
name: dokken
cap_add:
- NET_RAW
cap_drop
The cap_drop
configuration option allows you to specify additional system capabilities to remove from the container. See the Docker Runtime Privilege and Linux Capabilities Documentation for a complete list of capabilities.
driver:
name: dokken
cap_dop:
- AUDIT_WRITE
chef_image
The chef_image
configuration option allows you to specify Docker image other than the official Chef Infra Client image. This can be used to specify the Cinc Docker image or internal customer builds of Chef Infra Client.
driver:
name: dokken
chef_image: tas50/my_chef_fork
cgroupns_host
This configuration can be used to pass --cgroupns=host
flag during the docker run, which can be beneficial in the following scenario.
Docker Desktop 4.3.0+ uses cgroupv2 which breaks containers that use systemd
.
Containers that use systemd
need to run with --privileged --cgroupns=host -v /sys/fs/cgroup:/sys/fs/cgroup:rw
, which can be achieved with the following configuration.
driver:
name: dokken
privileged: true
cgroupns_host: true
volumes:
- /sys/fs/cgroup:/sys/fs/cgroup:rw
Reference: 4.3.0 Release notes
chef_version
The chef_version
configuration option allows you to specify which Docker image tag of the Chef Infra Client image to use. By default latest
is used, which is equivalent to the latest version the Chef’s stable
channel. For a complete list of available tags see chef/chef tags on Docker Hub.
driver:
name: dokken
chef_image: 17 # the latest 17.x release
creds_file
The creds_file
configuration options allows you to specify the location of a json
file that contains the authentication credentials for the private docker registries.
platforms:
- name: centos-7
driver:
image: reg/centos-7
creds_file: './creds.json'
The credentials file should contain all the necessary details that are required to authenticate the private registry. A sample credentials file is as follows.
{
"username": "org_username",
"password": "password",
"email": "[email protected]",
"serveraddress": "https://registry.org.com/"
}
deprecations_as_errors
The deprecations_as_errors
configuration option specifies that Test Kitchen should fail if any deprecations are encountered in Chef Infra cookbooks. This flag is useful when testing cookbooks in CI systems as it helps identify code that will later block upgrading Chef Infra Client.
driver:
name: dokken
deprecations_as_errors: true
dns
The dns
configuration option allows you to specify custom dns servers.
driver:
name: dokken
dns:
- 1.1.1.1
dns_search
The dns_search
configuration option allows you to specify custom dns search domains.
driver:
name: dokken
dns_search: example.com
docker_host_url
The docker_host_url
configuration option allows you to specify a Docker daemon other than localhost.
driver:
name: dokken
docker_host_url: tcp://10.0.1.49:2376
docker_registry
The docker_registry
configuration option allows you to use Docker registries other than Docker Hub including Docker Hub mirrors or private registries. If using a private registry make sure that registry has the Chef Infra image, any dokken images in your kitchen.yml
file, as well as the centos:7 image used during the converge process.
driver:
name: dokken
docker_registry: docker.example.com
env
Use the env
configuration option to set environment variables in the Test Kitchen container.
driver:
name: dokken
env:
- SOME_VARIABLE=value
hostname
The hostname
configuration allows you to set a custom hostname for the container. This configuration option can be used to set a hostname per suite to test clusters or client/server architectures. This is possible due to Docker’s internal DNS resolution which allows each container on the internal dokken
network to resolve the IP of one another. This configuration option defaults to dokken
---
driver:
name: dokken
provisioner:
name: dokken
transport:
name: dokken
platforms:
- name: ubuntu-20.04
driver:
image: dokken/ubuntu-20.04
suites:
- name: cluster-server-1
driver:
hostname: cluster-server-1
run_list:
- recipe[my_cookbook::default]
- name: cluster-server-2
driver:
hostname: cluster-server-2
run_list:
- recipe[my_cookbook::default]
$ kitchen login cluster-server-1-ubuntu-2004
root@cluster-server-1:/# ping cluster-server-2
PING cluster-server-2 (172.18.0.3) 56(84) bytes of data.
64 bytes from de87305be1-cluster-server-2-ubuntu-2004.dokken (172.18.0.3): icmp_seq=1 ttl=64 time=0.138 ms
hostname_aliases
The hostname_aliases
configuration option allows you to configure additional hostnames that will resolve to the container:
- name: cluster-server-1
driver:
hostname: cluster-server-1
hostname_aliases:
- also_this_hostname
image_prefix
The image_prefix
configuration option allows you to add a prefix to the name of all containers generated by Test Kitchen. This can be helpful for avoiding conflicts when multiple Test Kitchen instances are running against the same Docker daemon.
driver:
name: dokken
image_prefix: acme
intermediate_instructions
The intermediate_instructions
configuration option allows you to define steps to run on the Test Kitchen container before you converge Chef Infra Client. This is very useful for updating package caches on operation systems like Debian/Ubuntu or other preparation tasks that you might need to run.
platforms:
- name: ubuntu
driver:
image: ubuntu:20.04
intermediate_instructions:
- RUN apt-get update
- RUN apt-get -y install some_extra_package
pid_one_command: /lib/systemd/systemd
ipv6
The ipv6
configuration options enables IPv6 in the Docker daemon. This configuration option should be considered a global setting for all containers since dokken does not update the dokken network once it’s been created. It is not recommend to use this configuration option within suites.
driver:
name: dokken
ipv6: true
You can check to see if IPv6 is enabled on the dokken network by seeing if the following command returns true:
docker network inspect dokken --format='{{.EnableIPv6}}'
If the command returns false, we recommend you delete the network and allow dokken to recreate it with IPv6.
To allow IPv6 Docker networks to reach the internet IPv6 firewall rules must be set up. The simplest way to achieve this is to update Docker’s /etc/docker/daemon.json to use the following settings. You will need to restart the docker daemon after making these changes.
{
"experimental": true,
"ip6tables": true
}
Some containers require the ip6table_filter kernel module to be loaded on the host system or ip6tables will not dunction on the container (Centos 7 for example). To check if the module is loaded use the command
sudo lsmod | grep ip6table_filter
If there is no output then the module is not loaded and should be loaded using the command
modprobe ip6table_filter
ipv6_subnet
The ipv6_subnet
configuration option specifies the IPv6 subnet to run containers in. This configuration option requires that the ipv6
configuration option be set true. Similar to the ipv6
configuration option, this option should be considered global to all platform/suites since these network options aren’t reset between suites.
driver:
name: dokken
ipv6: true
ipv6_subnet: "2001:db8:1::/64" # "2001:db8::/32 Range reserved for documentation"
memory_limit
The memory_limit
configuration option specifies the maximum amount of memory that a Test Kitchen container can consume. By default the memory limit of the containers you run is unbound (or limited by the Docker client on macOS).
driver:
name: dokken
memory_limit: 2147483648 # 2GB
pid_one_command
The pid_one_command
configuration option sets the pid 1 command of the container being created. By default this is a simple loop command. For testing Chef Infra cookbooks that manage services you will need to set this to the init system binary.
platforms:
- name: ubuntu-20.04
driver:
image: dokken/ubuntu-20.04
pid_one_command: /bin/systemd
- name: centos-8
driver:
image: dokken/centos-8
pid_one_command: /usr/lib/systemd/systemd
platform
This options can be used to specify which platform/architecture needs to be used. It allows users to specify either the following in a global config file
---
driver:
platform: linux/amd64
or the following under a specif platform.
platforms:
- name: ubuntu-20.04
driver:
image: dokken/ubuntu-20.04
platform: linux/amd64
ports
The ports
configuration option specifies ports on the local container to map back to the host running Test Kitchen. This is useful when testing web services so you can browse them locally on your workstation.
suites:
- name: my_webserver
run_list:
- recipe[my_cookbook::default]
driver:
ports:
- 80 # map 80 on the container to 80 locally
- 9000:8080 # map 8080 on the container to 9000 locally
- 9001:8081/tcp # map TCP 8080 on the container to TCP 9000 locally
privileged
The privileged
configuration option allows starting the container with extended Docker privileges. This is necessary when testing systemd services in Chef Infra cookbooks. By default this value is false
.
driver:
name: dokken
privileged: true
pull_chef_image
The pull_chef_image
configuration option specifies if the Chef Infra docker image should be pulled or not.
driver:
name: dokken
pull_chef_image: false
pull_platform_image
The pull_platform_image
configuration option specifies if the Linux distro container should be pulled by Docker before each Test Kitchen converge. This option is set to true
by default, but it may be useful to disable it in low bandwidth situations or if you are testing local platform containers that aren’t yet pushed to Docker Hub.
driver:
name: dokken
pull_platform_image: false
tmpfs
The tmpfs
configuration option specifies tmpfs volumes to mount on the Test Kitchen container including the mount option for those volumes.
driver:
name: dokken
tmpfs:
/tmp: rw,noexec,nosuid,nodev,size=65536k
volumes
The volumes
configuration option specifies local volumes to mount into the Test Kitchen containers.
driver:
name: dokken
volumes:
- /mnt # mount the local /mnt directory to /mnt in the container
Provisioner Configuration Options
root_path
The root_path
configuration option specifies the directory used by Chef Infra Client for storing cookbooks, remote file downloads, and other cached data. By default /opt/kitchen
is used.
chef_binary
The chef_binary
configuration option specifies the path to the Chef Infra Client binary. By default kitchen-dokken uses /opt/chef/bin/chef-client
. If you’d like to test with Chef Infra Solo you may want to change this to /opt/chef/bin/chef-solo
.
chef_options
The chef_options
configuration option specifies a string of additional command line flags to pass to the Chef Infra Client. By default the -z
flag for running in Local Mode is specified.
chef_log_level
The chef_log_level
configuration option specifies the logging level used when running Chef Infra Client. By default this is set to warn
. info
, debug
, or trace
levels may also be specified.
chef_output_format
The chef_output_format
configuration option specifies the output formatter used by Chef Infra Client when logging. By default this is set to doc
for the documentation formatter. For less verbose output you may set this to minimal
.
chef_license
The chef_license
configuration option specified if the Chef Infra license should be automatically accepted in the Test Kitchen run. Valid values: accept
, accept-no-persist
, or accept-silent
provisioner:
chef_license: accept-no-persist
profile_ruby
The profile_ruby
configuration option specifies enabling low level Ruby profiling of the Chef Infra Client when run by Test Kitchen. This flag is used by Chef Infra Client contributors for testing of the application itself. For testing performance of cookbooks see slow_resource_report
instead.
slow_resource_report
The slow_resource_report
configuration option specifies running Chef Infra Client with the slow resource report mode enabled. This functionality is available in Chef Infra Client 17.2 or later and provides timing information on the 10 slowest resources in your cookbooks.
provisioner:
slow_resource_report: true
clean_dokken_sandbox
When Chef Infra Client converges kitchen-dokken populates /opt/kitchen/ with the Chef Infra Client and Test Kitchen data required to converge. By default this directory is cleared out at the end of every run. One of the subdirectories of /opt/kitchen/ is the chef cache directory. For cookbooks that download significant amounts of data from the network, i.e. many remote_file calls, this can make subsequent converges unnecessarily slow. If you would like the chef cache to be preserved between converges add clean_dokken_sandbox: false to the provisioner section of kitchen.yml. The default value is true.
provisioner:
name: dokken
clean_dokken_sandbox: false
Transport Configuration Options
timeout
The timeout
configuration option specifies the timeout in seconds for communicating with the Docker daemon.
Dokken Linux Containers
Specially created containers for kitchen-dokken, build off official Linux distro images, but include all of the packages and services necessary to test Chef Infra cookbooks. These containers are produced for leading Linux distributions such as CentOS, OpenSUSE, Amazon Linux, and Ubuntu. For a complete list of available dokken specific container images see u/dokken on Docker Hub.
Example kitchen.yml
---
driver:
name: dokken
privileged: true # allows systemd services to start
provisioner:
name: dokken
transport:
name: dokken
verifier:
name: inspec
platforms:
- name: ubuntu-20.04
driver:
image: dokken/ubuntu-20.04
pid_one_command: /bin/systemd
intermediate_instructions:
- RUN /usr/bin/apt-get update
- name: centos-8
driver:
image: dokken/centos-8
pid_one_command: /usr/lib/systemd/systemd
suites:
- name: default
run_list:
- recipe[my_cookbook::default]
verifier:
inspec_tests:
- test/integration/default
Using Kitchen Dokken with Podman
Using Dokken with podman is a little less straight forward than with Docker. The main problem is volumes are not populated when they are first created.
As per this issue, we can use lifecycle hooks to create the volume and populate it with the Chef executable before we try and start the main container.
Note, if you’re using a specific version of Chef, and not latest, then you need to reference the correct version in your podman create command because this breaks the automatic pulling of the correct version of the Chef Docker image by kitchen-dokken.
---
driver:
name: dokken
privileged: true # allows systemd services to start
provisioner:
name: dokken
transport:
name: dokken
verifier:
name: inspec
platforms:
- name: ubuntu-20.04
driver:
image: dokken/ubuntu-20.04
pid_one_command: /bin/systemd
intermediate_instructions:
- RUN /usr/bin/apt-get update
- name: centos-8
driver:
image: dokken/centos-8
pid_one_command: /usr/lib/systemd/systemd
suites:
- name: default
run_list:
- recipe[test_linux::default]
verifier:
inspec_tests:
- test/integration/default
lifecycle:
pre_create:
- podman create --name chef-latest --replace docker.io/chef/chef:latest sh
- podman start chef-latest
post_destroy:
- podman volume prune -f