Does splunk play nice with puppet?

[matt]

I'd like to able to install and configure the log forwarder using puppet. What needs to be done to make that happen?

[jpuppets]

This is an older thread but wanted to provide an updated answer to the question since it appears high on Google search results. There is a specific module to manage configuration of Splunk with Puppet. You can find out more information here: https://forge.puppet.com/puppet/splunk.

[b2c]

Here is another method on how to install the splunk universal forwarder (linux).
I looked into many puppet modules, but none actually solved the auto-accept license problem for me in an acceptable manner (like, not messing with init scripts or the like). So I hacked my own module. Our setup includes puppet-dashboard and an internal package repository in which the splunk forwarder packages are checked in, so we can install them with apt/yum.

I also added an option to remove the splunk forwarder from the servers in case it is no longer needed.

The installation is controlled via the variable 'splunk_uf_enable' which is set in puppet-dashboard. I guess the module is short enough to be easily adapted to work without puppet dashboard variables.

init.pp

class splunk_uf {
  if $::splunk_uf_enabled == 0 {
    notify { 'Splunk universal forwarder disabled in dashboard, undeploying installation and removing startup links. Configuration files will be preserved.': }
    include splunk_uf::undeploy
  }

  if $::splunk_uf_enabled == 1 {
    notify { 'Splunk universal forwarder enabled in dashboard, deploying installation and adding startup links.': }
    include splunk_uf::deploy
  }
}

deploy.pp

class splunk_uf::deploy {
  notify { 'Will deploy, configure and enable splunk forwarder.': }
  include splunk_uf::config
  include splunk_uf::service
  package { "splunkforwarder":
    ensure  =>      "present",
    require =>      Exec["aptitude_update"],
    before  =>      Class['splunk_uf::config', 'splunk_uf::service'],
  }
}

class splunk_uf::deploy {
  notify { 'Will deploy, configure and enable splunk forwarder.': }
  include splunk_uf::config
  include splunk_uf::service

  package { "splunkforwarder":
    ensure  => "present",
    require => Exec["aptitude_update"],
    before  => Class['splunk_uf::config', 'splunk_uf::service'],
  }
}

class splunk_uf::config {
  file { "/opt/splunkforwarder/etc/system/local/outputs.conf":
    content => template("splunk_uf/opt/splunkforwarder/etc/system/local/outputs.conf.erb"),
    mode  => "0644",
    owner => "root",
    group => "root",
    notify => Service["splunk"],
    require => Class['splunk_uf::deploy'],
  }
}

class splunk_uf::service {
  exec { "splunk_uf_enable_boot_start_accept_license":
    command => "/opt/splunkforwarder/bin/splunk enable boot-start --accept-license --no-prompt --answer-yes",
    onlyif => "/opt/splunkforwarder/bin/splunk enable boot-start --no-prompt 2>&1 | egrep -i '.*not.*accepted.*'",
    path => "/opt/splunk/bin:/usr/bin:/usr/sbin:/bin",
    logoutput => true,
  }

  service { "splunk":
    enable => true,
    ensure => "running",
    hasrestart => true,
    hasstatus => true,
    require => Class['splunk_uf::config'],
  }
}
class splunk_uf::config {
  file { "/opt/splunkforwarder/etc/system/local/outputs.conf":
    content =>  template("splunk_uf/opt/splunkforwarder/etc/system/local/outputs.conf.erb"),
    mode  => "0660",
    owner => "root",
    group => "root",
    notify => Service["splunk"],
    require => Class['splunk_uf::deploy'],
    }
}

class splunk_uf::service {
  exec { "splunk_uf_enable_boot_start_accept_license":
    command => "/opt/splunkforwarder/bin/splunk enable boot-start --accept-license --no-prompt --answer-yes",
    onlyif =>  /opt/splunkforwarder/bin/splunk enable boot-start --no-prompt 2>&1 | egrep -i '.*not.*accepted.*'",
    path => "/opt/splunk/bin:/usr/bin:/usr/sbin:/bin",
    logoutput =>  true,
  }

  service { "splunk":
    enable => true,
    ensure => "running",
    hasrestart => true,
    hasstatus  => true,
    require => Class['splunk_uf::config'],
  }
}

undeploy.pp

class splunk_uf::undeploy {
  notify { 'Will undeploy and disable splunk forwarder. Config will be preserved': }

  service { "splunk":
    enable      =>  false,
    ensure      =>  "stopped",
    hasrestart  =>  true,
    hasstatus   =>  true,
  }

  package { "splunkforwarder":
    ensure  =>  "absent",
  }
}

Minimal template: outputs.conf.erb

[tcpout]
  defaultGroup = indexers
[tcpout:indexers]
  server=splunk:9997

[softwareimprove]

Please have a look at my github project. We can deploy correctly universal forwarders and unix app to CentOS 6, Ubuntu 12 and Opensuse 12 (tested, also on EC2 instances). The module use a generated template, which can left to default or customized on a per-host basis. I'm open to suggestions and pull requests! My next release will include SSL.

[vincesesto]

Hello,
I just wanted to add to this conversation. I have had a lot of issues with setting up puppet to install and run splunkforwarders. It took me about a day to get the process down, and hope that the following description is able to help.

When we install the splunkforwaders on our environment we always ensure that we are running as the splunk user and run the following command at install time:
sudo /opt/splunkforwarder/bin/splunk enable boot-start -user -splunk

So to get our puppet manifests to run and install correctly we add the /etc/init.d/splunk script in our install module:

class splunkforwarder::install_splunkforwarder {
  file { '/etc/init.d/splunk':
    ensure  => 'present',
    owner   => 'root',
    group   => 'root',
    mode    => '0700',
    source  => "puppet:///modules/splunkforwarder/splunk_init",
  }
  package { 'splunkforwarder':
    ensure  => present,
    require => File['/etc/init.d/splunk'],
  }
}

The one change that we have made to the init.d file is to add the --accept-license as an extra argument to the start command like so:

/bin/su splunk -c "\"/opt/splunkforwarder/bin/splunk\" start --no-prompt --answer-yes --accept-license"

We have found this to work correctly on fresh installs and moving forward with any changes on splunkforwarders already running. It means that we do not have to worry about installing the license file as well.

[dgarstang]

Splunk does NOT work well with puppet. There are quite a number of files that are auto-generated by Splunk, and re-pushing out the old files with puppet will break Splunk. Also, if you deploy all the files you need with puppet, when you start Splunk for the first time, it thinks your doing an upgrade. Don't know what the repurcusions of that might be.

[infinitiguy]

it should be fine as long as you get your required order proper and make sure not to overwrite anything that would be changed during normal operations of splunk. I'm planning on writing a splunk module to deploy universal forwarders and I don't see any reason why it con't be done.

[lisa]

A more complete Splunk class for Puppet, to turn most nodes into lightweight forwarders (and one indexer), can be found at Github: http://bit.ly/dhvN36.

The class is based on hexx0's SSL tutorial.

[thartmann]

Here is my current (working) manifest! (v.2)

class splunk::forwarder { $mod = "splunk"

service {

  "splunk":
     ensure     => running,
     enable     => true,
     hasrestart => true,
     hasstatus  => false,
     pattern    => "splunkd",
     subscribe  => [
        Package["splunk"],
        File["/opt/splunk/etc/apps/unix"],
        File["/opt/splunk/etc/apps/SplunkLightForwarder"],
        File["/etc/init.d/splunk"]
     ]

}

package {

  "splunk":
      ensure  => installed,

}

file {

  "/opt/splunk/etc/splunk.license":
     mode    => "644",
     owner   => "splunk",
     group   => "splunk",
     backup  => true,
     ensure  => present,
     require => Package["splunk"],
     source  => "puppet:///modules/$mod/noarch/opt/splunk/etc/splunk-forwarder.license";

  "/etc/init.d/splunk":
     mode    => "700",
     owner   => "root",
     group   => "root",
     ensure  => present,
     require => Package["splunk"],
     source  => "puppet:///modules/$mod/noarch/etc/init.d/splunk";

  "/opt/splunk/etc/passwd":
     mode     => "600",
     owner    => "root",
     group    => "root",
     backup   => true,
     ensure   => present,
     require  => Package["splunk"],
     source   => "puppet:///modules/$mod/noarch/opt/splunk/etc/passwd";

  "/opt/splunk/etc/auth":
     owner   => "splunk",
     group   => "splunk",
     mode    => "600",
     recurse => true,
     purge   => false,
     require => Package["splunk"],
     source  => "puppet:///modules/$mod/noarch/opt/splunk/etc/auth";

  "/opt/splunk/etc/apps/unix":
     owner   => "splunk",
     group   => "splunk",
     recurse => true,
     purge   => false,
     require => Package["splunk"],
     source  => "puppet:///modules/$mod/noarch/opt/splunk/etc/apps/unix";

  "/opt/splunk/etc/apps/SplunkLightForwarder":
     owner   => "splunk",
     group   => "splunk",
     recurse => true,
     purge   => false,
     require => Package["splunk"],
     source  => "puppet:///modules/$mod/noarch/opt/splunk/etc/apps/SplunkLightForwarder",

}

}

[bizza]

I just working on a puppet config that didn't restart correctly the universal forwarder, because of exit status 0 on both start and stop splunk status.
hasstatus false and pattern splunkd is a nice information for me, thanks 🙂

[pde23]

When you start your forwarder, are you using

$SPLUNK_HOME/bin/splunk start --accept-license

to tell it not to give you the EULA?

[thartmann]

This is a great question! I'm writing a puppet manifest to deploy my forwarders based on this answer. However I can't seem to get the forwarder license right. I'm copying "splunk-forwarder.license" into splunk.license" but I still get a EULA on first start... is there something i'm missing? I'd be happy to post my manifest here as an example once I'm finished!

Thanks!

[thartmann]

Awesome! I think that will work well! I'll post my current manifest! Thanks much guys!

[gkanapathy]

as pde23 says, yeah, just edit the script and add the flags --accept-license --no-prompt --answer-yes to the default start command. Then just push out that script using puppet. Don't use have puppet execute ./splunk enable boot-start, that's not as nice. You could add the --accept-license etc flags to the command if you have puppet execute it, but I prefer just pushing files as it easier to specify a specific end-state that way than by running commands.

[thartmann]

I'm using just the init script (/etc/init.d/splunk start|stop|restart) to start the forwarder, I suppose I could edit the script after it's created, but I was hoping there was a better way to accept, or place the license. I'm trying create an automated deploy with puppet, for our standard builds.

[gkanapathy · Accepted solution]

Puppet works very well with Splunk. You should configure a Puppet package to push the Splunk installer out to the forwarder, along with some base installation items such as the forwarder license, a script to enable the service at boot, and a matched set of $SPLUNK_HOME/etc/auth/, $SPLUNK_HOME/etc/passwd, and $SPLUNK_HOME/etc/system/local/server.conf files to set a password for the forwarder's Splunk admin user. (You could also include a matched authorize.conf file for LDAP integration instead if desired.)

On top of that, you can then push the forwarder-specific configuration files and apps that are necessary to run as a forwarder.

If you have many classes of Splunk forwarder, or other classes of Splunk server, you can create separate packages for each one to push a different set of configurations. I do recommend that you organize the configuration sets using Splunk apps, and group the apps into Puppet packages by server class.