Puppet: managing zero-knowledge secrets

 Or: how to use /bin/true as a hinge of sorts

I was recently tasked with adding a zero-knowledge secrets module for our Puppet environment, using age and mustache. At first glance this seemed like a simple task, just decrypt the secrets and use mustache to drop them into the target files
This works great for single-secret files, but multi-secret files turned out to be a bit of a problem:

$ cat demo.pp
# core::age
#
# define: core::age
# this class takes no parameters, it only installs basic requirements to use core::age::decrypt
#
# if your {secrets,templates} are in a subdir of /home/user/.age/{secrets,templates} then you must define that subdir on your own
#
# README in files/age/README.md

class core::age::demo {
  $tmpdir      = '/tmp/age'
  ['age', 'ruby-mustache',].each | $package | {
    case $::kernel {
      'Darwin': {
        $pkg = $package ? {
          'ruby-mustache' => 'mustache',
          default  => $package,
        }
        $provider = $package ? {
          'ruby-mustache' => 'gem',
          default  => 'brew',
          }
        $ensure = 'present'
        }
      default: {
        $pkg      = $package
        $provider = 'apt'
        $ensure   = $package ? {
          # require age >=1.0.0 https://github.com/sthagen/FiloSottile-age/pull/22/files
          'age'   => '>=1.0.0-1~bpo11+1',
          default => 'present',
        }
      }
    }
    package {$pkg:
      ensure   => $ensure,
      provider => $provider,
    }
  }
  file {
    default:
      ensure  => directory,
      owner   => 'user',
      group   => 'user',
      mode    => '0700',
      # don't allow unknown files in these dirs, they're old secrets
      recurse => true,
      purge   => true,
      force   => true,
    ;
    "/home/user/.age":
      recurse => remote,
      source  => 'puppet:///modules/core/age',
    ;
    "/home/user/.age/config":
      # this is where the host keys live, puppet doesn't manage them
      purge   => false,
      recurse => false,
    ;
    $tmpdir:
      owner => root,
      group => roo,
  }
}
$ cat decryptdemo.pp
define core::age::decryptdemo (
  Hash                      $secrets,
  String                    $template,
  Enum['present', 'absent'] $ensure          = present,
  String                    $target          = $name,
  Enum[
    'cluster',
    'host'
  ]                         $key_type        = 'cluster',
  String                    $template_source = "puppet:///modules/core/age/templates/${template}",
  Boolean                   $plant_template  = true,
  String                    $mode            = '0600',
  Optional[String]          $owner           = undef,
  Optional[String]          $group           = undef,
) {

  include core::age::demo

  $random        = generate('/bin/bash', '-c', 'tr -dc A-Za-z0-9  '/usr/bin/true',
    default  => '/bin/true',
  }
  $false          = $::kernel ? {
    'Darwin' => '/usr/bin/false',
    default  => '/bin/false',
  }

  # fail if can't find the key, run max 1x per key type
  if ! defined(Exec["ensure ${key_type} key"]) {
    exec {"ensure ${key_type} key":
      command     => $false,
      unless      => "test -f ${keypath}",
      refreshonly => true,
    }
  }

  if $plant_template {
    if ! defined (File[$template_path]) {
      file {$template_path:
        ensure => core::bool2ensure($plant_template),
        owner  => 'user',
        group  => 'user',
        mode   => '0600',
        source => $template_source,
      }
    }
  }

  if $ensure == 'present' {
    $secrets.each | $reference, $secret | {
      $secret_path = "/home/user/.age/secrets/${secret}.age"
      file {$secret_path:
        owner  => 'user',
        group  => 'user',
        mode   => '0600',
        source => "puppet:///modules/core/age/secrets/${secret}.age",
        notify => Exec["create_${target}",],
      }
      ~> exec {"fail_if_cannot_decrypt_${secret}":
        # errors get hidden in decrypt_${secret} by the $(command substitution)
        command     => "age --decrypt -i ${keypath} ${secret_path} > /dev/null",
        path        => "/usr/local/bin:/bin:/usr/bin",
        subscribe   => File[$secret_path,],
        logoutput   => false,
        refreshonly => true,
      }
      ~> exec{"decrypt_${secret}":
        command     => "echo ${reference}: \"$(age --decrypt -i ${keypath} ${secret_path})\" >> ${tmpfile}",
        path        => "/usr/local/bin:/bin:/usr/bin",
        subscribe   => Exec["fail_if_cannot_decrypt_${secret}",],
        notify      => Exec["create_${target}",],
        require     => [File[$secret_path,],],
        refreshonly => true,
      }
    }
    exec {"create_${target}":
      command     => "mustache ${tmpfile} ${template_path} >| ${target}",
      logoutput   => false,
      path        => "/usr/local/bin:/bin:/usr/bin",
      # require $target bc otherwise puppet thinks the file doesn't exist
      # (bc it doesn't at the start of the first puppet run) and overwrites it blank
      require     => [File[$template_path, $target,],],
      refreshonly => true,
    }
  }
  file {$target:
    # file resource is only here to apply ownership & perms
    ensure => $ensure,
    owner  => $owner,
    group  => $group,
    mode   => $mode,
  }
}
$ cat ../../files/age/templates/demo_test
this is a test file
the secret is: {{{ demo_test }}}
the secret is above

$ cat ../../../../manifests/site.pp
node /^demo[0-9].demo.net$/
{
  core::age::decryptdemo {'/home/user/demo_test':
    ensure   => present,
    secrets  => {
      'demo_test' => 'demo_test',
    },
    template => 'demo_test',
  }
}

$ echo test | age -a -R secrets/cluster/demo.pub -o secrets/puppet/demo_test.age

user@demo1:~$ sudo puppet agent -t --environment demo
Notice: /Stage[main]/Main/Node[__node_regexp__demo0-9.demo.net]/Core::Age::Decryptdemo[/home/user/demo_test]/File[/home/user/.age/secrets/demo_test.age]/ensure: defined content as '{sha256}579c545e1cbea83fe5dc5d69c36e2cbe9ed43ac12b3886202b16386e4441ac44'
Info: /Stage[main]/Main/Node[__node_regexp__demo0-9.demo.net]/Core::Age::Decryptdemo[/home/user/demo_test]/File[/home/user/.age/secrets/demo_test.age]: Scheduling refresh of Exec[create_/home/user/demo_test]
Info: /Stage[main]/Main/Node[__node_regexp__demo0-9.demo.net]/Core::Age::Decryptdemo[/home/user/demo_test]/File[/home/user/.age/secrets/demo_test.age]: Scheduling refresh of Exec[fail_if_cannot_decrypt_demo_test]
Info: /Stage[main]/Main/Node[__node_regexp__demo0-9.demo.net]/Core::Age::Decryptdemo[/home/user/demo_test]/File[/home/user/.age/secrets/demo_test.age]: Scheduling refresh of Exec[fail_if_cannot_decrypt_demo_test]
Notice: /Stage[main]/Main/Node[__node_regexp__demo0-9.demo.net]/Core::Age::Decryptdemo[/home/user/demo_test]/Exec[fail_if_cannot_decrypt_demo_test]: Triggered 'refresh' from 2 events
Info: /Stage[main]/Main/Node[__node_regexp__demo0-9.demo.net]/Core::Age::Decryptdemo[/home/user/demo_test]/Exec[fail_if_cannot_decrypt_demo_test]: Scheduling refresh of Exec[decrypt_demo_test]
Info: /Stage[main]/Main/Node[__node_regexp__demo0-9.demo.net]/Core::Age::Decryptdemo[/home/user/demo_test]/Exec[fail_if_cannot_decrypt_demo_test]: Scheduling refresh of Exec[decrypt_demo_test]
Notice: /Stage[main]/Main/Node[__node_regexp__demo0-9.demo.net]/Core::Age::Decryptdemo[/home/user/demo_test]/Exec[decrypt_demo_test]: Triggered 'refresh' from 2 events
Info: /Stage[main]/Main/Node[__node_regexp__demo0-9.demo.net]/Core::Age::Decryptdemo[/home/user/demo_test]/Exec[decrypt_demo_test]: Scheduling refresh of Exec[create_/home/user/demo_test]
Notice: /Stage[main]/Main/Node[__node_regexp__demo0-9.demo.net]/Core::Age::Decryptdemo[/home/user/demo_test]/Exec[create_/home/user/demo_test]: Triggered 'refresh' from 2 events
Notice: Applied catalog in 9.65 seconds
user@demo1:~$ cat /home/user/demo_test
this is a test file
the secret is: test
the secret is above

This looks great! But things get tricky if you add a second secret, or change one secret but not the other:

$ echo test2 | age -a -R secrets/cluster/demo.pub -o secrets/puppet/demo_test2.age

$ cat ../../../../manifests/site.pp
node /^demo[0-9].demo.net$/
{
  core::age::decryptdemo {'/home/user/demo_test':
    ensure   => present,
    secrets  => {
      'demo_test' => 'demo_test',
      'demo_test2' => 'demo_test2',
    },
    template => 'demo_test',
  }
}

user@demo1:~$ sudo puppet agent -t --environment demo
Notice: /Stage[main]/Main/Node[__node_regexp__demo0-9.demo.net]/Core::Age::Decryptdemo[/home/user/demo_test]/File[/home/user/.age/templates/demo_test]/content:
--- /home/user/.age/templates/demo_test        2022-09-02 20:14:43.003075691 +0000
+++ /tmp/puppet-file20220902-1060326-16onh8q    2022-09-02 20:23:47.190767817 +0000
@@ -1,3 +1,6 @@
 this is a test file
 the secret is: {{{ demo_test }}}
 the secret is above
+secret 2 is below
+secret2 is: {{{ demo_test2 }}}
+secret 2 is above

Notice: /Stage[main]/Main/Node[__node_regexp__demo0-9.demo.net]/Core::Age::Decryptdemo[/home/user/demo_test]/File[/home/user/.age/templates/demo_test]/content: content changed '{sha256}f1247e0b81a9a0f7899a7aa342001033728142fa89af54cc81fe3fab8e784ff8' to '{sha256}54dea21440b9c83833fcf35c2ffcadf2e7134ef2a62a9eecba56f12658ba0168'
Notice: /Stage[main]/Main/Node[__node_regexp__demo0-9.demo.net]/Core::Age::Decryptdemo[/home/user/demo_test]/File[/home/user/.age/secrets/demo_test2.age]/ensure: defined content as '{sha256}4a930d67df8ee0af7e5a7c7574427e1b8a363fe20f534a3f0eb79a659bf07d7e'
Info: /Stage[main]/Main/Node[__node_regexp__demo0-9.demo.net]/Core::Age::Decryptdemo[/home/user/demo_test]/File[/home/user/.age/secrets/demo_test2.age]: Scheduling refresh of Exec[create_/home/user/demo_test]
Info: /Stage[main]/Main/Node[__node_regexp__demo0-9.demo.net]/Core::Age::Decryptdemo[/home/user/demo_test]/File[/home/user/.age/secrets/demo_test2.age]: Scheduling refresh of Exec[fail_if_cannot_decrypt_demo_test2]
Info: /Stage[main]/Main/Node[__node_regexp__demo0-9.demo.net]/Core::Age::Decryptdemo[/home/user/demo_test]/File[/home/user/.age/secrets/demo_test2.age]: Scheduling refresh of Exec[fail_if_cannot_decrypt_demo_test2]
Notice: /Stage[main]/Main/Node[__node_regexp__demo0-9.demo.net]/Core::Age::Decryptdemo[/home/user/demo_test]/Exec[fail_if_cannot_decrypt_demo_test2]: Triggered 'refresh' from 2 events
Info: /Stage[main]/Main/Node[__node_regexp__demo0-9.demo.net]/Core::Age::Decryptdemo[/home/user/demo_test]/Exec[fail_if_cannot_decrypt_demo_test2]: Scheduling refresh of Exec[decrypt_demo_test2]
Info: /Stage[main]/Main/Node[__node_regexp__demo0-9.demo.net]/Core::Age::Decryptdemo[/home/user/demo_test]/Exec[fail_if_cannot_decrypt_demo_test2]: Scheduling refresh of Exec[decrypt_demo_test2]
Notice: /Stage[main]/Main/Node[__node_regexp__demo0-9.demo.net]/Core::Age::Decryptdemo[/home/user/demo_test]/Exec[decrypt_demo_test2]: Triggered 'refresh' from 2 events
Info: /Stage[main]/Main/Node[__node_regexp__demo0-9.demo.net]/Core::Age::Decryptdemo[/home/user/demo_test]/Exec[decrypt_demo_test2]: Scheduling refresh of Exec[create_/home/user/demo_test]
Notice: /Stage[main]/Main/Node[__node_regexp__demo0-9.demo.net]/Core::Age::Decryptdemo[/home/user/demo_test]/Exec[create_/home/user/demo_test]: Triggered 'refresh' from 2 events
Notice: Applied catalog in 9.90 seconds
user@demo1:~$ sudo cat /home/user/demo_test
this is a test file
the secret is:
the secret is above
secret 2 is below
secret2 is: test2
secret 2 is above
user@demo1:~$

As you can see, we've placed secret2 but we've lost the original secret

In order to work around this, I used /bin/true as a 'hinge' of sorts between the File resources and the Exec resources:

$ cat base.pp decrypt.pp secret.pp
# core::age
#
# define: core::age
# this class takes no parameters, it only installs basic requirements to use core::age::decrypt
#
# if your {secrets,templates} are in a subdir of /home/user/.age/{secrets,templates} then you must define that subdir on your own
#
# README in files/age/README.md

class core::age::base {
  $tmpdir      = '/tmp/age'
  ['age', 'ruby-mustache',].each | $package | {
    case $::kernel {
      'Darwin': {
        $pkg = $package ? {
          'ruby-mustache' => 'mustache',
          default  => $package,
        }
        $provider = $package ? {
          'ruby-mustache' => 'gem',
          default  => 'brew',
          }
        $ensure = 'present'
        }
      default: {
        $pkg      = $package
        $provider = 'apt'
        $ensure   = $package ? {
          # require age >=1.0.0 https://github.com/sthagen/FiloSottile-age/pull/22/files
          'age'   => '>=1.0.0-1~bpo11+1',
          default => 'present',
        }
      }
    }
    package {$pkg:
      ensure   => $ensure,
      provider => $provider,
    }
  }
  file {
    default:
      ensure  => directory,
      owner   => 'user',
      group   => 'user',
      mode    => '0700',
      # don't allow unknown files in these dirs, they're old secrets
      recurse => true,
      purge   => true,
      force   => true,
    ;
    "/home/user/.age":
      recurse => remote,
      source  => 'puppet:///modules/core/age',
    ;
    "/home/user/.age/config":
      # this is where the host keys live, puppet doesn't manage them
      purge   => false,
      recurse => false,
    ;
    $tmpdir:
      owner => root,
      group => root,
  }
}
# define: core::age::decrypt
#
# README in files/age/README.md
#
# Parameters:
# [*ensure*]          - Ensure that the files is (present|absent). Default: present
# [*secrets*]         - A hash of {'references' => 'secrets'}. Required
# [*template*]        - A file template that will be placed on the target host, for age+mustache to fill. Required
#                     - This is not the final file, but a mustache template that age+mustache will interact with to build the final target
# [*target*]          - The target file to be created. Default: $name.
# [*key_type*]        - The key type, choose between {cluster,host}. Default: cluster
# [*template_source*] - The template source file. Default: puppet:///modules/core/age/templates/${template}. Required.
# [*plant_template*]  - If you want puppet to automatically source the template file using $template_source. Default: true
#                     - Set to false in order to fill a template file with other (non-secret) vars using a file resource + epp
# [*mode*]            - The file mode to be set on the target. Default: '0600'. Optional
# [*user*]            - The file owner to be set on the target. Default: undef. Optional
# [*group*]           - The file group to be set on the target. Default: undef. Optional
#
# A note about the use of ` > /dev/null` in many of the `exec` statements in this file
# The purpose of this module is to _not_ leak secrets.
# By default an exec statment will push its results back to the puppet catalog, even when `logoutput => false,`
# Hence we use `> /dev/null` to keep the secrets from being placed on the puppet catalog
#
# Because this is a zero-knowledge module, puppet can't know if the content of your target file has changed outside of this module
# The only thing that will update the file is
# - removing the target file
# - removing the corresponding secret or template in ~/.age/{secrets,templates}
# - changing the secret in puppet
# simply changing the content of the target file will not trigger an update of the secrets
#
# or you can wait one hour and an automatic refresh of the secrets will be performed

define core::age::decrypt (
  Hash                      $secrets,
  String                    $template,
  Enum['present', 'absent'] $ensure          = present,
  String                    $target          = $name,
  Enum[
    'cluster',
    'host'
  ]                         $key_type        = 'cluster',
  String                    $template_source = "puppet:///modules/core/age/templates/${template}",
  Boolean                   $plant_template  = true,
  String                    $mode            = '0600',
  Optional[String]          $owner           = undef,
  Optional[String]          $group           = undef,
) {

  include core::age::base

  $random        = generate('/bin/bash', '-c', 'tr -dc A-Za-z0-9  '/usr/bin/true',
    default  => '/bin/true',
  }
  $false          = $::kernel ? {
    'Darwin' => '/usr/bin/false',
    default  => '/bin/false',
  }

  exec {"kickoff_${target}":
    # a harmless kicker-offer that every file notifies and every exec subscribes to
    # this is the entrypoint to any secret
    command     => $true,
    refreshonly => true,
  }

  # fail if can't find the key, run max 1x per key type
  if ! defined(Exec["ensure ${key_type} key"]) {
    exec {"ensure ${key_type} key":
      command     => $false,
      unless      => "test -f ${keypath}",
      refreshonly => true,
      subscribe   => Exec["kickoff_${target}",],
    }
  }

  if $plant_template {
    if ! defined (File[$template_path]) {
      file {$template_path:
        ensure => core::bool2ensure($plant_template),
        owner  => 'user',
        group  => 'user',
        mode   => '0600',
        source => $template_source,
        notify => Exec["kickoff_${target}",],
      }
    }
  }

  if $ensure == 'present' {
    $secrets.each | $reference, $secret | {
      core::age::secret {$secret:
        reference => $reference,
        tmpfile   => $tmpfile,
        target    => $target,
        keypath   => $keypath,
        tag       => $::hostname,
      }
    }
    # collect all the secrets before working on decryption
    Core::Age::Secret <<| tag == $::hostname |>>
    ~> exec {"create_${target}":
      command     => "mustache ${tmpfile} ${template_path} >| ${target}",
      logoutput   => false,
      path        => "/usr/local/bin:/bin:/usr/bin",
      # require $target bc otherwise puppet thinks the file doesn't exist
      # (bc it doesn't at the start of the first puppet run) and overwrites it blank
      require     => [File[$template_path, $target,],],
      subscribe   => Exec["kickoff_${target}",],
      refreshonly => true,
    }
    exec {"hourly_refresh_${target}":
      # allows ${target} to be refreshed by other changes (as often as necessary) or regularly on a schedule
      command  => $true,
      notify   => Exec["kickoff_${target}",],
      schedule => hourly,
    }
  }
  file {$target:
    # file resource is only here to apply ownership &amp; perms
    ensure => $ensure,
    owner  => $owner,
    group  => $group,
    mode   => $mode,
    notify => Exec["kickoff_${target}",],
  }
}
# core::age::secret

define core::age::secret (
  $reference,
  $tmpfile,
  $target,
  $keypath,
  $secret     = $name,
) {

  $secret_path = "/home/user/.age/secrets/${secret}.age"
  file {$secret_path:
    owner  => 'user',
    group  => 'user',
    mode   => '0600',
    source => "puppet:///modules/core/age/secrets/${secret}.age",
    notify => Exec["kickoff_${target}",],
  }
  ~> exec {"fail_if_cannot_decrypt_${secret}":
    # errors get hidden in decrypt_${secret} by the $(command substitution)
    command     => "age --decrypt -i ${keypath} ${secret_path} > /dev/null",
    path        => ":/usr/local/bin:/bin:/usr/bin",
    subscribe   => Exec["kickoff_${target}",],
    logoutput   => false,
    refreshonly => true,
  }
  ~> exec{"decrypt_${secret}":
    command     => "echo ${reference}: \"$(age --decrypt -i ${keypath} ${secret_path})\" >> ${tmpfile}",
    path        => "/usr/local/bin:/bin:/usr/bin",
    subscribe   => Exec["kickoff_${target}",],
    notify      => Exec["create_${target}",],
    require     => [File[$secret_path,],],
    refreshonly => true,
  }
}
node /^demo[0-9].demo.net$/
{
  core::age::decrypt {'/home/user/demo_test':
    ensure   => present,
    secrets  => {
      'demo_test'  => 'demo_test',
    },
    template => 'demo_test',
  }
}
user@demo4:~$ sudo puppet agent -t --environment damon
Notice: /Stage[main]/Main/Node[__node_regexp__demo0-9.demo.net]/Core::Age::Decrypt[/home/user/demo_test]/File[/home/user/demo_test]/ensure: created (corrective)
Info: /Stage[main]/Main/Node[__node_regexp__demo0-9.demo.net]/Core::Age::Decrypt[/home/user/demo_test]/File[/home/user/demo_test]: Scheduling refresh of Exec[kickoff_/home/user/demo_test]
Notice: /Stage[main]/Main/Node[__node_regexp__demo0-9.demo.net]/Core::Age::Decrypt[/home/user/demo_test]/Exec[kickoff_/home/user/demo_test]: Triggered 'refresh' from 1 event
Info: /Stage[main]/Main/Node[__node_regexp__demo0-9.demo.net]/Core::Age::Decrypt[/home/user/demo_test]/Exec[kickoff_/home/user/demo_test]: Scheduling refresh of Exec[create_/home/user/demo_test]
Info: /Stage[main]/Main/Node[__node_regexp__demo0-9.demo.net]/Core::Age::Decrypt[/home/user/demo_test]/Exec[kickoff_/home/user/demo_test]: Scheduling refresh of Exec[fail_if_cannot_decrypt_demo_test]
Info: /Stage[main]/Main/Node[__node_regexp__demo0-9.demo.net]/Core::Age::Decrypt[/home/user/demo_test]/Exec[kickoff_/home/user/demo_test]: Scheduling refresh of Exec[decrypt_demo_test]
Notice: /Stage[main]/Main/Node[__node_regexp__demo0-9.demo.net]/Core::Age::Decrypt[/home/user/demo_test]/Core::Age::Secret[demo_test]/Exec[fail_if_cannot_decrypt_demo_test]: Triggered 'refresh' from 1 event
Info: /Stage[main]/Main/Node[__node_regexp__demo0-9.demo.net]/Core::Age::Decrypt[/home/user/demo_test]/Core::Age::Secret[demo_test]/Exec[fail_if_cannot_decrypt_demo_test]: Scheduling refresh of Exec[decrypt_demo_test]
Notice: /Stage[main]/Main/Node[__node_regexp__demo0-9.demo.net]/Core::Age::Decrypt[/home/user/demo_test]/Core::Age::Secret[demo_test]/Exec[decrypt_demo_test]: Triggered 'refresh' from 2 events
Info: /Stage[main]/Main/Node[__node_regexp__demo0-9.demo.net]/Core::Age::Decrypt[/home/user/demo_test]/Core::Age::Secret[demo_test]/Exec[decrypt_demo_test]: Scheduling refresh of Exec[create_/home/user/demo_test]
Notice: /Stage[main]/Main/Node[__node_regexp__demo0-9.demo.net]/Core::Age::Decrypt[/home/user/demo_test]/Exec[create_/home/user/demo_test]: Triggered 'refresh' from 2 events
Notice: Applied catalog in 10.44 seconds
user@demo4:~$ sudo cat /home/user/demo_test
this is a test file
the secret is: test
the secret is above
secret 2 is below
secret2 is:
secret 2 is above

node /^demo[0-9].demo.net$/
{
  core::age::decrypt {'/home/user/demo_test':
    ensure   => present,
    secrets  => {
      'demo_test'  => 'demo_test',
      'demo_test2'  => 'demo_test2',
    },
    template => 'demo_test',
  }
}

user@demo4:~$ sudo puppet agent -t --environment damon
Notice: /Stage[main]/Main/Node[__node_regexp__demo0-9.demo.net]/Core::Age::Decrypt[/home/user/demo_test]/Core::Age::Secret[demo_test2]/File[/home/user/.age/secrets/demo_test2.age]/mode: mode changed '0700' to '0600'
Info: /Stage[main]/Main/Node[__node_regexp__demo0-9.demo.net]/Core::Age::Decrypt[/home/user/demo_test]/Core::Age::Secret[demo_test2]/File[/home/user/.age/secrets/demo_test2.age]: Scheduling refresh of Exec[kickoff_/home/user/demo_test]
Info: /Stage[main]/Main/Node[__node_regexp__demo0-9.demo.net]/Core::Age::Decrypt[/home/user/demo_test]/Core::Age::Secret[demo_test2]/File[/home/user/.age/secrets/demo_test2.age]: Scheduling refresh of Exec[fail_if_cannot_decrypt_demo_test2]
Notice: /Stage[main]/Main/Node[__node_regexp__demo0-9.demo.net]/Core::Age::Decrypt[/home/user/demo_test]/Exec[kickoff_/home/user/demo_test]: Triggered 'refresh' from 1 event
Info: /Stage[main]/Main/Node[__node_regexp__demo0-9.demo.net]/Core::Age::Decrypt[/home/user/demo_test]/Exec[kickoff_/home/user/demo_test]: Scheduling refresh of Exec[create_/home/user/demo_test]
Info: /Stage[main]/Main/Node[__node_regexp__demo0-9.demo.net]/Core::Age::Decrypt[/home/user/demo_test]/Exec[kickoff_/home/user/demo_test]: Scheduling refresh of Exec[fail_if_cannot_decrypt_demo_test]
Info: /Stage[main]/Main/Node[__node_regexp__demo0-9.demo.net]/Core::Age::Decrypt[/home/user/demo_test]/Exec[kickoff_/home/user/demo_test]: Scheduling refresh of Exec[decrypt_demo_test]
Info: /Stage[main]/Main/Node[__node_regexp__demo0-9.demo.net]/Core::Age::Decrypt[/home/user/demo_test]/Exec[kickoff_/home/user/demo_test]: Scheduling refresh of Exec[fail_if_cannot_decrypt_demo_test2]
Info: /Stage[main]/Main/Node[__node_regexp__demo0-9.demo.net]/Core::Age::Decrypt[/home/user/demo_test]/Exec[kickoff_/home/user/demo_test]: Scheduling refresh of Exec[decrypt_demo_test2]
Notice: /Stage[main]/Main/Node[__node_regexp__demo0-9.demo.net]/Core::Age::Decrypt[/home/user/demo_test]/Core::Age::Secret[demo_test]/Exec[fail_if_cannot_decrypt_demo_test]: Triggered 'refresh' from 1 event
Info: /Stage[main]/Main/Node[__node_regexp__demo0-9.demo.net]/Core::Age::Decrypt[/home/user/demo_test]/Core::Age::Secret[demo_test]/Exec[fail_if_cannot_decrypt_demo_test]: Scheduling refresh of Exec[decrypt_demo_test]
Notice: /Stage[main]/Main/Node[__node_regexp__demo0-9.demo.net]/Core::Age::Decrypt[/home/user/demo_test]/Core::Age::Secret[demo_test]/Exec[decrypt_demo_test]: Triggered 'refresh' from 2 events
Info: /Stage[main]/Main/Node[__node_regexp__demo0-9.demo.net]/Core::Age::Decrypt[/home/user/demo_test]/Core::Age::Secret[demo_test]/Exec[decrypt_demo_test]: Scheduling refresh of Exec[create_/home/user/demo_test]
Notice: /Stage[main]/Main/Node[__node_regexp__demo0-9.demo.net]/Core::Age::Decrypt[/home/user/demo_test]/Core::Age::Secret[demo_test2]/Exec[fail_if_cannot_decrypt_demo_test2]: Triggered 'refresh' from 2 events
Info: /Stage[main]/Main/Node[__node_regexp__demo0-9.demo.net]/Core::Age::Decrypt[/home/user/demo_test]/Core::Age::Secret[demo_test2]/Exec[fail_if_cannot_decrypt_demo_test2]: Scheduling refresh of Exec[decrypt_demo_test2]
Notice: /Stage[main]/Main/Node[__node_regexp__demo0-9.demo.net]/Core::Age::Decrypt[/home/user/demo_test]/Core::Age::Secret[demo_test2]/Exec[decrypt_demo_test2]: Triggered 'refresh' from 2 events
Info: /Stage[main]/Main/Node[__node_regexp__demo0-9.demo.net]/Core::Age::Decrypt[/home/user/demo_test]/Core::Age::Secret[demo_test2]/Exec[decrypt_demo_test2]: Scheduling refresh of Exec[create_/home/user/demo_test]
Notice: /Stage[main]/Main/Node[__node_regexp__demo0-9.demo.net]/Core::Age::Decrypt[/home/user/demo_test]/Exec[create_/home/user/demo_test]: Triggered 'refresh' from 3 events
Notice: Applied catalog in 10.53 seconds
user@demo4:~$ sudo cat /home/user/demo_test
this is a test file
the secret is: test
the secret is above
secret 2 is below
secret2 is: test2
secret 2 is above
user@demo4:~$

As you can see, we have successfully added a secret to our manifest but convinced Puppet to decrypt all secrets, and only then run mustache to drop the secrets in the file
This is acheived by having every File resource notify our 'kicker-offer' and every Exec resource subscribe to the 'kicker-offer', and by moving secrets to its own defined type that all get collected before the decrypt step runs

Puppet: using Facts in an Exec clause (onlyif or unless)

 According to this issue on Puppet labs, there's simply no way to use facts in an Puppet Exec clause for the purpose of onlyif or unless, as such:

exec {'some_exec':
  command => '/usr/bin/cmd',
  # this `onlyif` will cause your manifest to fail
  onlyif       => $::facts[some_fact] == true,
} 

However I found a simple workaround for this, simply set a var with your facts to either 'true' or 'false' (string values) and use those in your `onlyif` block

if $::facts[some_fact] == true {
  $exec_true = 'true'
}
else {$exec_true = 'false'}

then use that in your exec block:

exec {'some_exec':
  command => '/usr/bin/cmd',
  # literally calls '/usr/bin/{true,false}'
  onlyif       => "/usr/bin/${exec_true},
} 

Deploying ESXi 6.0u3 via PXE in a lab

In my never ending quest for automation, I thought I'd experiment with what is required to deploy ESXi over the network to new servers.

In order to script this, and know that I hit every checkmark, I decided to write it first using Vagrant.

I wrote this because other posts on the subject are close, but not exactly spot-on. This post, using Vagrant, is guaranteed because it's built from scratch every time I deploy it. I destroyed and rebuilt the entire environment multiple times while creating this post.

It uses these packages:

dhcp                 x86_64    12:4.1.1-63.P1.el6.centos

syslinux             x86_64    4.04-3.el6

tftp-server          x86_64    0.49-8.el6

If those packages change, this tutorial might become out of date. You should be able to force a version of each package to install in the bootstrap.sh file.

This tutorial uses CentOS 6.4 64-bit as a DHCP server and Vagrant.

All the code needed for this tutorial is available at https://github.com/volvo64/VMwarePXE

You'll also need an ESXi ISO. I used VMware-VMvisor-Installer-6.0.0.update03-5050593.x86_64.iso. This should be in the same directory as your Vagrantfile and bootstrap.sh. Make sure to change references to this ISO in bootstrap.sh.

Sorry for the weird formatting, Blogger isn't the best platform to write about code on. Review the code on Github for that.

First, in Vagrantfile, we're going to specify Centos64:

config.vm.box = "forumone/centos64-64"

We also need to specify a bootstrap file:

config.vm.provision :shell, path: "bootstrap.sh"

Lastly, in the Vagrantfile we need to specify a private network to perform DHCP on:

config.vm.network "private_network", ip: "192.168.33.10"

192.168.33.0/24 works for me, you should use what works for you.

That's all for the Vagrantfile.

bootstrap.sh actually builds the services once the Vagrant box has been created. We need a few programs installed. Let's start with dhcp, tftp-server and syslinux:

#!/usr/bin/env bash

yum -y install dhcp tftp-server syslinux

Next we need the contents of the ESXi ISO available in our tftpboot directory. We can mount it from the /vagrant/ directory that's automatically shared from your host:

mount -o loop /vagrant/VMware-VMvisor-Installer-6.0.0.update03-5050593.x86_64.iso /mnt/

Copy the files:

cp -rf /mnt /var/lib/tftpboot/esxi60u3

And finally unmount the ISO:

umount /mnt/

Now that we have DHCP installed and the ISO copied we need to create the DHCP options. This block writes the options to /etc/dhcp/dhcpd.conf:

echo '# dhcpd.conf

#

# Sample configuration file for ISC dhcpd

#

# option definitions common to all supported networks...

option domain-name "damon.local";

option domain-name-servers localhost.localhost;

default-lease-time 600;

max-lease-time 7200;

# If this DHCP server is the official DHCP server for the local

# network, the authoritative directive should be uncommented.

authoritative;

# Use this to send dhcp log messages to a different log file (you also

# have to hack syslog.conf to complete the redirection).

log-facility local7;

allow booting;

allow bootp;

option client-system-arch code 93 = unsigned integer 16;

# This is a very basic subnet declaration.

subnet 192.168.33.0 netmask 255.255.255.0 {

range 192.168.33.100 192.168.33.110;

option routers 192.168.33.10;

}

class "pxeclients" {

match if substring(option vendor-class-identifier, 0, 9) = "PXEClient";

# specifies the TFTP Server

next-server 192.168.33.10;

if option client-system-arch = 00:07 or option client-system-arch = 00:09 {

# PXE over EFI firmware

filename = "esxi60u3/mboot.efi";

} else {

# PXE over BIOS firmware

filename = "pxelinux.0";

}

}

' > /etc/dhcp/dhcpd.conf

 We need to do the same for the TFTP options in /etc/xinetd.d/tftp:

echo 'service tftp

{

socket_type = dgram

protocol = udp

wait = yes

user = root

server = /usr/sbin/in.tftpd

server_args = -s /var/lib/tftpboot

disable = no

per_source = 11

cps = 100 2

flags = IPv4

}' > /etc/xinetd.d/tftp

 The original boot.cfg that we copied from the ESXi ISO contains references to "/" all over the place that are no longer valid. We need to remove them. To do that:

sed -i 's/\///g' /var/lib/tftpboot/esxi60u3/boot.cfg

We need to create some directories and move some files to make PXE boot:

mkdir -p /var/lib/tftpboot/pxelinux.cfg

cp /usr/share/syslinux/pxelinux.0 /var/lib/tftpboot/

echo y | cp /usr/share/syslinux/menu.c32 /var/lib/tftpboot/esxi60u3/ 

 The last change we need is to write the menu.c32 file:

echo 'DEFAULT esxi60u3/menu.c32

MENU TITLE ESXi-6.0 Boot Menu

NOHALT 1

PROMPT 0

TIMEOUT 300

LABEL install

KERNEL esxi60u3/mboot.c32

APPEND -c esxi60u3/boot.cfg

MENU LABEL ESXi-6.0U3 ^Installer

LABEL hddboot

LOCALBOOT 0x80

MENU LABEL ^Boot from local disk' > /var/lib/tftpboot/pxelinux.cfg/default

And finally restart some services:

/etc/init.d/xinetd restart

/etc/init.d/dhcpd restart

/etc/init.d/iptables stop

 /etc/init.d/ip6tables stop

Run 'vagrant up' and you should have a server after a few minutes!

To verify the network interfaces, check out its network config. You should see two NICs attached:

Adapter 1 is your management interface, Adapter 2 is your internal interface.

Create a new VM in Virtualbox. Give it at least 4GB of RAM and a little storage. Make sure the network is attached to the same network as Adapter 2:

In order to boot it you need to assign 2 CPUs:

Unfortunately since Virtualbox doesn't allow for Nested Virtualization (Nested VT-x/AMD-V) we won't actually be able to run anything on these hypervisors.

Start up your test VM, use F12 and boot it from the LAN and it will show you the ESXi installer:

After this point just install ESXi as normal and it should load up fine!

Thanks to:

https://www.virtuallyghetto.com/2015/10/support-for-uefi-pxe-boot-introduced-in-esxi-6-0.html

http://www.vstellar.com/2017/07/25/automating-esxi-deployment-using-pxe-boot-and-kickstart/

Will investigate stateless ESXi/running a real config directly from PXE soon, as well as upgrading to newer ESXi releases via PXE.

Password Security and you

Some of my clients approach me regarding potentially breached accounts, so I thought it best to explain why good password policies are essential in 2016.

The website https://haveibeenpwned.com lists nearly 2 billion breached accounts from 168 websites. These are breaches that have been released publicly, meaning that they're probably only a small fraction of total breached accounts in existence. In reality, breaches that have not been made known publicly, or haven't even been discovered likely comprise at least as many as are known. With this in mind, the only safe mindset to use today is to assume that your username and password have been compromised, and to implement good password hygiene to limit the damage that can be done.

A good password:

• is long
• is not shared between different services
• contains a mixture of upper and lowercase letters, numbers, and symbols
• is not a dictionary word
• has no personal meaning to the user (i.e. 'F1@c0' is a bad password, because it refers to my dog's name)
• doesn't follow a recognizable pattern (i.e. Cuenca2015 is bad, because I might try Cuenca2014 or Cuenca2016 on other sites).

Lastpass helps us accomplish the goals here. Lastpass is a program akin to saved passwords in Chrome, or the piece of paper on which you record all of your usernames and passwords. However, unlike both of those, Lastpass is more secure. Lastpass helps you generate completely random passwords for each and every site- that way, if your Paypal password is leaked, that same password won't work on your bank account. Lastpass is also more securable in that you can have it log out after a certain period of time, forcing a master password re-entry before logins are made available again.

I encourage you to look through https://lastpass.com/how-it-works/ to see how the program works. Also, they've recently made mobile use free, so I encourage you to install Lastpass on your phones as well as computers.

Once you start down this road, you'll quickly find yourself fixing 20 years of bad password policy. This process is not going to be easy. You can expect to spend a year finding old accounts and changing old passwords to be more secure. It's important though to start with the high-risk accounts- financial and email (and Facebook), so that if your Linkedin account is breached, it won't affect your bank accounts.

I expect my clients to have issues getting this started, but once you wrap your head around the way Lastpass works, I think you'll rest a little easier knowing that your accounts are safer.

If you are in need of a security refresher, feel free to contact me at damon@damonbreeden.com or message 419-210-3631 (US) or 099-033-0345 (EC, Whatsapp).