Test Kitchen Blog

Test Kitchen 1.4.0 Release Notes

by Fletcher Nichol on

For immediate release: Test Kitchen 1.4.0 is available on RubyGems.

  Update 2015-05-08
  The installation section has been updated with a link to the ChefDK version that ships Test Kitchen 1.4.0

Sitting down? Good, this could take a few minutes to read through. Today we're releasing version 1.4.0 of Test Kitchen with a little something for everyone. We have an aggressively reworked plugin system including 2 new plugin concepts (Transport and Verifiers), improved SSH resiliency when instances are booting, first steps at SSH compression to make converges quicker, better HTTP proxy support, a fully realized Windows story, and more.

Let's dig in a little, shall we?

Installing

If you use the ChefDK project, then upgrade to version 0.5.0 or later to get all the goodies!

If you have a Ruby workflow with RubyGems and/or Bundler this won’t be too much work:

Gem install with:

gem install test-kitchen --version "~> 1.4"
gem install kitchen-vagrant --version "~> 0.17"
gem install winrm-transport # only needed if you want to spin up Windows instances

To use in a project with Bundler, the following goes into your Gemfile:

gem "test-kitchen", "~> 1.4"
gem "kitchen-vagrant", "~> 0.17"
gem "winrm-transport" # only needed if you want to spin up Windows instances

Windows!

Note that this release has the much-fabled "Windows guest support". How do you get started? At the moment getting Vagrant base box images of Windows is still a bit of a pain, but if you have access to one, here are the versions that should "just work":

  • Windows Server 2012r2
  • Windows Server 2012r2 Core (you will need .NET 4.5 installed however)
  • Windows Server 2012 (your system/image must have the KB2842230 hotfix applied (https://support.microsoft.com/en-us/kb/2842230))
  • Windows Server 2008r2
  • Windows 8.1 Pro,Enterprise,etc.
  • Windows 8 Pro,Enterprise,etc. (your system/image must have the KB2842230 hotfix applied (https://support.microsoft.com/en-us/kb/2842230))
  • Windows 7

For a more in-depth example of getting started with Test Kitchen and Windows check out the Windows Test Flight with Vagrant blog post.

Let’s take a peek at some of the other highlights…

Other Highlights

Self-Aware Provisioners

This feature introduces a #call(state) method that exists in Kitchen::Provisioner::Base which will be invoked by Test Kitchen when the converge action is performed. For backwards compatibility, the same convergence "template" is used, relying on a small number of public methods that return command strings and 3 methods responsible for sandbox creation and cleanup.

The high-level description of the default #call(state) method is as follows:

  1. Create the temporary sandbox on the workstation with "stuff" to transfer to the remote instance.
  2. Run the #install_command on the remote instance, if it is implemented.
  3. Run the #init_command on the remote instance, if it is implemented.
  4. Transfer all files in the sandbox path to the remote instance's configured :root_path.
  5. Run the #prepare_command on the remote instance, if it is implemented.
  6. Run the #run_command on the remote instance, if it is implemented.

As a Provisioner author, you may elect to overwrite or partially re-implement the #call(state) method to do whatever you need in whatever order makes sense. This key difference allows Provisioner authors to entirely own the kitchen converge action and not rely on the Driver.

(Potentially Breaking) Provisioners responsible for converge action

This is potentially breaking to Driver authors if all of the following are true:

  • Your Driver currently directly inherits from Kitchen::Driver::Base
  • Your Driver implements/overrides the #converge method

Put another way, your Driver may have issues if it looks like the following:

module Kitchen
  module Driver
    class MyDriver < Kitchen::Driver::Base
      def converge(state)
        # custom converge work
      end
    end
  end
end

For the vast majority of open source Drivers in the wild, current behavior is maintained as they all inherit from Kitchen::Driver::SSHBase. This class has been cemented to preserve its current behavior, and Test Kitchen will invoke the #converge method for these Drivers.

A future deprecation process may remove the SSHBase backwards compatibility, but not without plenty of lead time and warning. Due to the constraints of semantic versioning, by definition, this wouldn't occur before a 2.x codebase release.

(Potentially Breaking) Transports responsible for login action

This is potentially breaking to Driver authors if all of the following are true:

  • Your Driver currently inherits from Kitchen::Driver::Base
  • Your Driver implements/overrides the #login_command method

Put another way, your Driver may have issues if it looks like the following:

module Kitchen
  module Driver
    class MyDriver < Kitchen::Driver::Base
      def login_command(state)
        # custom converge work
      end
    end
  end
end

For the vast majority of open source Drivers in the wild, current behavior is maintained as they all inherit from Kitchen::Driver::SSHBase. This class has been cemented to preserve its current behavior, and Test Kitchen will invoke the #login_command method for these Drivers.

Self-Aware Verifiers

This feature introduces a #call(state) method that exists in Kitchen::Verifier::Base which will be invoked by Test Kitchen when the verify action is performed. The setup action which previously installed the Busser gem and plugins, becomes a dummy or "no-op" action. In other words all previous behavior in the setup action now takes place in the verify action. For backwards compatibility, the same verify "template" is used, relying on a small number of public methods that return strings and 3 new methods responsible for sandbox creation and cleanup (with a very similar implementation to that in Provisioners).

The high-level description of the default #call(state) method is as follows:

  1. Create the temporary sandbox on the workstation with "stuff" to transfer to the remote instance.
  2. Run the #install_command on the remote instance, if it is implemented.
  3. Run the #init_command on the remote instance, if it is implemented.
  4. Transfer all files in the sandbox path to the Verifier's configured :root_path on the remote instance.
  5. Run the #prepare_command on the remote instance, if it is implemented.
  6. Run the #run_command on the remote instance, if it is implemented.

As a Verifier author, you may elect to overwrite or partially re-implement the #call(state) method to do whatever you need in whatever order makes sense. This key difference allows Verifier authors to entirely own the kitchen verify action and not rely on the Driver.

(Potentially Breaking) Verifiers responsible for verify action

This is a potentially breaking change to Driver authors if all of the following are true:

  • Your Driver currently directly inherits from Kitchen::Driver::Base
  • Your Driver implements/overrides the #setup and/or #verify methods

Put another way, your Driver may have issues if it looks like the following:

module Kitchen
  module Driver
    class MyDriver < Kitchen::Driver::Base
      def setup(state)
        # custom setup work
      end

      def verify(state)
        # custom verify work
      end
    end
  end
end

For the vast majority of open source Drivers in the wild, current behavior is maintained as they all inherit from Kitchen::Driver::SSHBase. This class has been cemented to preserve its current behavior, and Test Kitchen will invoke the #setup and #verify methods for these Drivers.

Add Platform :os_type for instance path type hinting

A new configurable attribute is introduced to a Platform entry called :os_type. For example:

---
driver:
  name: docker

provisioner:
  name: chef_zero

platforms:
  - name: ubuntu-14.04
    os_type: unix
  - name: windows-8.1
    os_type: windows

The interpretation of :os_type is very narrowly defined as follows:

  • "windows" means a Windows operating system, requiring Windows paths such as "C:\\Users", "$env:TEMP\\stuff", or "%TEMP%\\more".
  • "unix" means a non-Windows operating system, or UNIX derivative–the implication is essentially the same. This implies unix-style paths such as "some/path/" or "/absolute/paths/are/nice".
  • nil or unset will default to meaning "unix" to ensure backwards compatibility if values are somehow not properly passed in.
  • Any other value will be passed down into the system, and allows for some future operating system support or a flag for bizarre custom behavior. This part is where dragons live.

Add remote host :shell_type hinting support

Platform has a new method #shell_type which will normally return either "powershell" or "bourne" depending on the pre-declared capabilities of the remote instance. The implicit default will be "bourne" for backwards compatibility.

Add :sudo_command to Provisioners, Verifiers, & ShellOut

In an effort to better support older distributions (such as CentOS 5), and other distributions which don't ship with sudo in $PATH (such as Solaris), a new configuration attribute is introduced into Provisioner::Base and Verifier::Base called :sudo_command. For greatest portability and backwards compatibility this defaults to "sudo -E" but is now customizable depending on your situation.

For example:

---
driver:
  name: vagrant

platforms:
  - name: centos-7.0                      # defaults apply
  - name: centos-5.11                     # removes -E flag
    provisioner:
      sudo_command: sudo
    verifier:
      sudo_command: sudo
  - name: solaris-10                      # sets custom path to sudo
    provisioner:
      sudo_command: /usr/local/bin/sudo
    verifier:
      sudo_command: /usr/local/bin/sudo

Note that a future feature proposes a way to remove the provisioner/verifier duplication but this would only be a convenience, not a behavior change.

Finally, Driver::Base was not augmented with this configuration attribute as it is no longer responsible for creating the commands to execute on remote instances–this is now solely the purview of Provisioner and Verifier plugins.

Add plugin diagnostics, exposed via kitchen diagnose

This feature adds a new flag to kitchen diagnose: --plugins which adds a :plugins hash to the diagnostic output. It is the unique set of all loaded Driver, Provisioner, Verifier, and Transport plugins which is keyed by the #name of the plugin.

For example, a .kitchen.yml with the following configuration:

---
driver:
  name: vagrant

provisioner:
  name: chef_zero

verifier:
  name: dummy

platforms:
  - name: ubuntu-14.04
    transport:
      name: ssh
  - name: centos-7.0
    driver:
      name: docker
    transport:
      name: ssh
  - name: windows-2012r2-core
    transport:
      name: winrm

with an invocation of kitchen diagnose --plugins would produce a diagnostic output similar to:

---
timestamp: 2015-03-28 00:43:32 UTC
kitchen_version: 1.4.0.dev
plugins:
  driver:
    Docker:
      class: Kitchen::Driver::Docker
    Vagrant:
      class: Kitchen::Driver::Vagrant
  provisioner:
    ChefZero:
      class: Kitchen::Provisioner::ChefZero
  transport:
    Ssh:
      class: Kitchen::Transport::Ssh
    Winrm:
      class: Kitchen::Transport::Winrm
  verifier:
    Dummy:
      class: Kitchen::Verifier::Dummy

Add :keepalive & :keepalive_interval attributes to SSH Transport

By default, keepalive packets are enabled with a 60-second interval. Both of these settings are configurable in a transport block, such as:

---
driver:
  name: vagrant

provisioner:
  name: chef_zero

transport:
  name: ssh
  keepalive: true
  keepalive_interval: 5

platforms:
  - name: ubuntu-14.04

suites:
  - name: default

Add default :compression & :compression_level attributes to SSH Transport

There are 2 new configuration attributes for all SSH Transports:

  • :compression - can be set to "zlib" or "none", default is "zlib"
  • :compression_level - can be set to a number between 0 and 9 where 0 is uncompressed and 9 is maximum compression, default is 9

These values are passed directly into the net-ssh library when being invoked.

Increased HTTP proxy support for executing commands

This release also added more a complete strategy for HTTP proxy variable setting. If both :http_proxy and :https_proxy are set, then the following Bourne shell environment variables are exported:

  • http_proxy
  • HTTP_PROXY
  • https_proxy
  • HTTPS_PROXY

And the following PowerShell environment variables are set:

  • $env:http_proxy
  • $env:HTTP_PROXY
  • $env:https_proxy
  • $env:HTTPS_PROXY

These environment variables will be set for every command executed in the Chef-related and Shell Provisioners, as well as the Busser Verifier.

The Bourne shell environment variable setting has also changed. Previously they were set with an env prepended to the sh -c '...' wrapped command. Now these environment variables are set inside the sh -c '...' at the top and are exported.

For example running wget "http://chef.io/chef/install.sh" with :http_proxy and :https_proxy set would generate a command similar to:

sh -c '
http_proxy="http://proxy"; export http_proxy
HTTP_PROXY="http://proxy"; export HTTP_PROXY
https_proxy="https://proxy"; export https_proxy
HTTPS_PROXY="https://proxy"; export HTTPS_PROXY

wget "http://chef.io/chef/install.sh"
'

Support symbol values in solo.rb (chef_solo) & client.rb (chef_zero)

This allows for settings such as the following to serialize correctly into client.rb or solo.rb (depending on the Provisioner). For example:

---
driver:
  name: vagrant

provisioner:
  name: chef_zero
  client_rb:
    ssl_verify_mode: :verify_none

platforms:
  - name: centos-7.0

suites:
  - name: default

Add :chef_metadata_url to Chef Provisioners for Windows installs

This configuration attribute is only used for Windows-based platforms and computes a suitable default based on:

  • The base of the value of :chef_omnibus_url
  • The version information in :require_chef_omnibus
  • Any project info given in :chef_omnibus_install_options

For example, given the following YAML fragment:

---
provisioner:
  name: chef_zero
  require_chef_omnibus: 12.1

platforms:
  - name: windows-8

would result in the latest 12.1.x Chef Omnibus package to be installed.

Here's an example installing the ChefDK package instead of a vanilla Chef Omnibus package:

---
provisioner:
  name: chef_zero
  require_chef_omnibus: 0.4
  chef_omnibus_install_options: -P chefdk

platforms:
  - name: windows-8
    provisioner:
      chef_omnibus_root: $env:systemdrive\opscode\chefdk

Full Changeleog

(This is a selected roll-up of 1.4.0 pre-release changelogs)

Potentially breaking changes

Note:: while a huge amount of effort has gone into preserving backwards compatibility, there could be issues when running this release using certain Drivers and Provisioners, especially ones that are deeply customized. Drivers that inherit directly from Kitchen::Driver::Base may need to be updated, while Driver that inherit directly from Kitchen::Driver::SSHBase should continue to operate as before. Other libraries/addons/plugins which patch internals of Test Kitchen's code may break or work differently and would be extremely hard to preserve while adding new functionality. Sadly, this is a tradeoff.

  • Drivers are no longer responsible for converge, setup, verify, and login actions. The updated Driver API contract (Driver::Base) only requires implementing the #create and #destroy methods, same as before. However, for Drivers that directly inherit from Kitchen::Driver::Base, any custom #converge, #setup, #verify, or #login_command methods will no longer be called. (@fnichol)
  • Drivers which inherit directly from Kitchen::Driver::SSHBase are now considered "Legacy Drivers" as further improvements for these Drivers may not be available in future releases. The previous behavior is preserved, i.e. the Driver's #converge, #setup, and #verify methods are called and all methods signatures (and relative behavior) is preserved. (Driver::SSHBase, Commit notes) (@fnichol)
  • Provisioners are now self-aware, completely owning the converge action. The original public methods of the Base Provisioner are maintained but are now invoked with a #call(state) method on the Provisioner object. Provisioner authors may elect to implement the command and sandbox methods, or re-implement the #call method which may not call any of the previously mentioned methods. (Provisioner::Base, Commit notes) (@fnichol)
  • Transport are not responsible for the login command. (Commit notes) (@fnichol)
  • Busser is now a plugin of type Verifier (see below for details on Verifiers). Any external code that directly creates a Kitchen::Busser object will fail as the class has moved to Kitchen::Verifier::Busser. Any external code that directly invokes Busser's #sync_cmd will log a warning and will not transfer test files (authors of plugins may now call instance.transport(state).upload(locals, remote) in its place). (@fnichol)
  • Verifiers are responsible for the verify action. (Commit notes) (@fnichol)
  • Pull request #649: Preserve Busser's #setup_cmd, #run_cmd, & #sync_cmd for better backwards compatibility. (@fnichol)
  • Pull request #672: Extract WinRM-dependant code from Transport::Winrm into the winrm-transport gem, meaning that WinRM support is now a soft dependency of Test Kitchen, similar to Berkshelf and Librarian-Chef. This means the first time a Winrm Transport is requested, a kitchen command will crash with a UserError message instructing the user to install the winrm-transport gem. Existing projects which do not use the Winrm Transport will be unaffected and have no extra gem dependencies to manage. (@fnichol)

Bug fixes

  • Issue #611, pull request #673: Ensure that secret key is deleted before converge for chef_zero and chef_solo Provisioners. (@fnichol)
  • Issue #389, pull request #674: Expand path for :ssh_key if provided in kitchen.yml for Ssh Transport. (@fnichol)
  • Pull request #653: Consider :require_chef_omnibus = 11 to be a modern version for Chef Provisioners. (@fnichol)

New features

  • ChefZero Provisioner supports Windows paths and PowerShell commands and works with the WinRM Transport (default behavior for Platform names starting with /^win/). (Provisioner::ChefZero) (@fnichol)
  • ChefSolo Provisioner supports Windows paths and PowerShell commands and works with the WinRM Transport (default behavior for Platform names starting with /^win/). (Provisioner::ChefSolo) (@fnichol)
  • Shell Provisioner supports PowerShell scripts in addition to Bourne shell scripts (Provisioner::Shell) (@fnichol)
  • Platform operating system and shell hinting: By default, Windows platform names (case insensitive platform names starting with /^win/) will have :os_type set to "windows" and :shell_type set to "powershell". By default, non-Windows platform names will have :os_type set to "unix" and :shell_type set to "bourne". The methods #windows_os?, #unix_os?, #powershell_shell?, #bourne_shell?, and #remote_path_join are available for all Driver, Provisioner, Verifier, and Transport authors. (@fnichol)
  • New plugin type: Transport, which executes commands and transfers files to remote instances. (Transport::Base) (@afiune, @mwrock, @fnichol)
  • New Transport: WinRM: which re-uses a remote shell to execute commands and upload files over WinRM. Currently non-SSL/plaintext authentication only. (Transport::Winrm) (@afiune, @mwrock, @fnichol)
  • New Transport: SSH, which re-uses one SSH connection where possible. Improvements such as keepalive, retries, and further configuration attributes are included. This replaces the more general Kitchen:SSH class, which remains in the codebase for plugins that call this class directly. (Transport::Ssh) (@fnichol)
  • New plugin type: Verifier, which executes post-convergence tests on the instance. Busser is now a Verifier. (Verifier::Base) (@fnichol)
  • Add API versioning metadata to all plugin types. (@fnichol)
  • Pull request #667, pull request #668: Add plugin diagnostics, exposed via kitchen diagnose. (@fnichol)
  • Pull request #675, issue #424: Add default :compression & :compression_level configuration attributes to Ssh Transport.
  • Pull request #651, issue #592, issue #629, issue #307: Add :sudo_command to Provisioners, Verifiers, & ShellOut. (@fnichol)

Improvements

  • In addition to supporting setting http_proxy and https_proxy environment variables when :http_proxy and :https_proxy are set in Provisioner and Verifier blocks, HTTP_PROXY and HTTPS_PROXY environment variables will also be set/exported in ChefZero/ChefSolo Provisioners and Busser Verifier. (@fnichol)
  • Pull request #600, pull request #633, issue #85: Add --log-overwrite flag to CLI anywhere --log-level is accepted. By default it is true and will clear out the log every time Test Kitchen runs. To disable this behavior pass --log-overwrite=false or --no-log-overwrite. You can also configure this with the environment variable KITCHEN_LOG_OVERWRITE. (@tyler-ball)
  • Refactor "non-trivial" (i.e. more than a line or two) Bourne and PowerShell code bodies into static files under support/ for better code review by domain experts. (@fnichol)
  • Pull request #530, issue #429: Stop uploading empty directories. (@whiteley)
  • Pull request #588: Change getchef.com to chef.io in ChefZero and ChefSolo Provisioners. (@jdmundrawala)
  • Pull request #658, issue #654: Updated for sh compatibility based on install.sh code which supports more platforms including Solaris. (@scotthain, @curiositycasualty, @fnichol)
  • Pull request #652, pull request #666, issue #556: Support symbol values in solo.rb & client.rb for chef_zero and chef_solo Provisioners. (@fnichol)
Made in North America

© 2013, Heavy Water Operations, LLC (OR).