Splunk App for Infrastructure deploy through deployment server

[StefanW]

Hello,

we are trying out the Splunk App for Infrastructure. With the easy install script for windows, we get the data in. Fine. But we dont want to run the script on every universal forwarder manually. Because of this, we want to deploy the inputs.conf through our deployment server. But in every inputs.conf are an _meta option where the dimensions are defined. 

Has someone a solution to deploy the configuration through deployment server and also set the dimension options right ?

[eddieddieddie]

Hi Stefan,

I've faced the same problem trying to deploy the Splunk App for Infrastructure to Windows via a deployment server - Splunk provide no "all Splunk" way of doing it. I think they expect you will be using another tool to deploy your forwarders and config (something like Ansible or SCCM I suppose). However I worked out another way to deploy this config which does use the deployment server and then set the metadata on a per-machine basis via a scripted input with the "interval" configured to -1 (run only once).

Create a new app on your deployment server - for example "Windows-Base-Monitoring".

Under this create two folders: 'local' and 'bin'.

In the bin folder create a PowerShell script to open the inputs.conf file, look for stanza's titled "[perfmon://...]" and then update the _meta field with info specific to that host. The following script does this and inserts a _meta line containing the hosts OS version, IP address, hostname.

## InjectMetaConfigs.ps1

# Collect the hosts meta data essentials:
# extract os & ip info
$os_info = Get-CimInstance Win32_OperatingSystem | Select-Object Caption, Version
$ip_info = Test-Connection -ComputerName $env:computername -count 1 | Select-Object IPV4Address

# Add OS & ip info as dimension
# If these dimensions are to be updated, user has to rerun this script
$metadata = "_meta = os::" + "`"" + $os_info.Caption + "`""
$metadata = $metadata + " os_version::" + $os_info.Version
$metadata = $metadata + " ip::" + "`"" + $ip_info.IPV4Address.IPAddressToString + "`""
$metadata = $metadata + " entity_type::Windows_Host"

# Path to the Perfmon collection TA app:
# ($SplunkHome is defined by Splunk before this script is run.)
$perfmonApp = $SplunkHome + "/etc/apps/Windows-Base-Monitoring"
# The path to the input.conf file to be edited.
$inputsConfLoc = $perfmonApp + "/local/inputs.conf"

$regex = '(?<=\[perfmon\:\/\/\w+\])'
$inputfile = Get-Content ($inputsConfLoc) -Raw 
$outputfile = [regex]::replace($inputfile, '(?<=\[perfmon\:\/\/\w+\])(\r\n|\n)(?!_meta)', ($1 + "`n" + $metadata +"`n"))
Set-Content -path $inputsConfLoc -Value $outputfile

Now you can go ahead and create an inputs.conf file in the local folder with all the Perfmon configs set for everything you want to collect - however without the "_meta" lines as these will be set by the PowerScript after it's been deployed to the universal forwarders.  In this file also define a PowerShell scripted input  which runs the above script once only (interval = -1). For example:

# inputs.conf

# The following powershell script is run just once per restart (internal = -1) and if it finds the Perfmon stanza's below 
# doesn't contain '_meta' statements immidately after the [perfmon..] line adds it, completing the system info for the 
# it's running from.
[powershell://InjectMetaConfig]
script = . "$SplunkHome\etc\apps\Windows-Base-Monitoring\bin\InjectMetaConfigs.ps1"
interval = -1
disabled = false

### Perfom monitoring

## CPU
[perfmon://CPU]
counters = % C1 Time;% C2 Time;% Idle Time;% Processor Time;% User Time;% Privileged Time;% Reserved Time;% Interrupt Time;
disabled = false
instances = _Total
mode = single
interval = 60
object = Processor
useEnglishOnly = true
sourcetype = PerfmonMetrics:CPU
index = metrics

## Memory
[perfmon://Memory]
counters = Cache Bytes;% Committed Bytes In Use;Page Reads/sec;Pages Input/sec;Pages Output/sec;Committed Bytes;Available Bytes;
disabled = false
interval = 60
object = Memory
useEnglishOnly=true
sourcetype = PerfmonMetrics:Memory
mode = single
index = metrics

 When you deploy this app to your universal forwarders the deployment server handles sending the configuration and PowerShell script out. The UF then restarts itself and reads the new inputs.conf config, as it does this it runs the PowerShell script which updates the inputs.conf file with the _meta lines.  You might need to restart the UF's after this has happened, however in my experience I discovered the inputs.conf is updated before the Perfmon monitoring starts.

Hopefully this provides some inspiration to fixing your deployment!

Eddie