In the Splunk App for Infrastructure, can you use existing universal forwarders without running the script and reinstalling on all of our servers?

[omprakash9998]

Hi,

We have a Splunk environment with universal forwarders already installed on our Windows servers. We want to try the Splunk App for infrastructure. Can we use the existing Universal Forwarders to use the Splunk App for Infrastructure without having to running the script and reinstalling on all of our servers.

Thanks.

[myfriendhenry]

Here is a script that you can use. I use it to roll this out using Forwarder Management - but you can just run it on each host if you don't have too many. I named the add-on "ManagedInfrastructure" on the Univ Forwarders, but you can keep the original name just change that line in the script. If you want to use your Forwarder Manager, I have included that after the script.

install.ps1

$splunkpath = "C:\Program Files\SplunkUniversalForwarder\etc\apps\ManagedInfrastructure\local\" $inputconfTemp = $splunkpath + "inputs.temp"

  $metrics_param = "logical_disk,physical_disk,cpu,memory,network,system,process"
  $log_sources_param = "application%WinEventLog,system%WinEventLog,security%WinEventLog"
  $cpu_instance_type = '_Total'
  $metrics_index = "em_metrics"

if (Test-Path -Path $inputconfTemp ) {Remove-Item -Path $inputconfTemp}


$m_cpu =`
"[perfmon://CPU]`r`n" `
+ "counters = % C1 Time;% C2 Time;% Idle Time;% Processor Time;% User Time;% Privileged Time;% Reserved Time;% Interrupt Time`r`n" `
+ "instances = $cpu_instance_type`r`n" `
+ "interval = 30`r`n" `
+ "object = Processor`r`n" `
+ "mode = single`r`n" `
+ "useEnglishOnly = true`r`n" `
+ "index = $metrics_index"

$m_memory =`
"[perfmon://Memory]`r`n" `
+ "counters = Cache Bytes;% Committed Bytes In Use;Page Reads/sec;Pages Input/sec;Pages Output/sec;Committed Bytes;Available  
Bytes`r`n"`
+ "interval = 30`r`n" `
+ "object = Memory`r`n" `
+ "mode = single`r`n" `
+ "useEnglishOnly = true`r`n" `
+ "index = $metrics_index"

$m_physical_disk =`
"[perfmon://PhysicalDisk]`r`n" `
+ "counters = % Disk Read Time;% Disk Write Time`r`n" `
+ "instances = *`r`n" `
+ "interval = 30`r`n" `
+ "object = PhysicalDisk`r`n" `
+ "mode = single`r`n" `
+ "useEnglishOnly = true`r`n" `
+ "index = $metrics_index"

$m_logical_disk =`
"[perfmon://LogicalDisk]`r`n" `
+ "counters = Free Megabytes;% Free Space`r`n" `
+ "instances = *`r`n" `
+ "interval = 30`r`n" `
+ "object = LogicalDisk`r`n" `
+ "mode = single`r`n" `
+ "useEnglishOnly = true`r`n" `
+ "index = $metrics_index"

$m_network =`
"[perfmon://Network]`r`n" `
+ "counters = Bytes Received/sec;Bytes Sent/sec;Packets Received/sec;Packets Sent/sec;Packets Received Errors;Packets Outbound  
Errors`r`n" `
+ "instances = *`r`n" `
+ "interval = 30`r`n" `
+ "mode = single`r`n" `
+ "object = Network Interface`r`n" `
+ "useEnglishOnly = true`r`n" `
+ "index = $metrics_index"

$m_system =`
"[perfmon://System]`r`n" `
+ "counters = Processor Queue Length;Threads`r`n" `
+ "instances = *`r`n" `
+ "interval = 30`r`n" `
+ "object = System`r`n" `
+ "mode = single`r`n" `
+ "useEnglishOnly = true`r`n" `
+ "index = $metrics_index"

$m_process =`
"[perfmon://Process]`r`n" `
+ "counters = % Processor Time;% User Time;% Privileged Time`r`n" `
+ "instances = *`r`n" `
+ "interval = 30`r`n" `
+ "object = Process`r`n" `
+ "mode = single`r`n" `
+ "useEnglishOnly = true`r`n" `
+ "index = $metrics_index"

# Inputs.conf options for the supported Windows EventLogs
$eventlog_options =`
"checkpointInterval = 10`r`n" `
+ "current_only = 1`r`n" `
+ "disabled = 0`r`n" `
+ "start_from = oldest"  

# extract os & ip info
# > WIN 7
#$os_info = Get-CimInstance Win32_OperatingSystem | Select-Object Caption, Version

# =WIN 7
$os_info = Get-WmiObject Win32_OperatingSystem | Select-Object Caption, Version
$ip_info = Test-Connection -ComputerName $env:computername -count 1 | Select-Object IPV4Address

# Add OS & ip info as dimension
# If these dimensions are to be updated, user has to rerun this script
$dims = $dimensions + " os::" + "`"" + $os_info.Caption + "`""
$dims = $dims + " os_version::" + $os_info.Version
$dims = $dims + " ip::" + "`"" + $ip_info.IPV4Address.IPAddressToString + "`""
$dims = $dims + ' entity_type::Windows_Host'

$metrics = $metrics_param -split ','
$log_sources = $log_sources_param -split ','

echo "# *** Configure Metrics Logs collected ***" > $inputconfTemp


For ($i=0; $i -lt $metrics.Length; $i++) {
  $m_name = "m_" + $metrics[$i]
  Get-Variable -Name $m_name -ValueOnly -ErrorAction 'Ignore' >> $inputconfTemp
  # Add dimensions
  echo "_meta = $dims" >> $inputconfTemp
  echo "`n" >> $inputconfTemp
}


For ($i=0; $i -lt $log_sources.Length; $i++) {
  if([string]::IsNullOrEmpty($log_sources[$i])) { continue }
  # split log source into source and sourcetype
  $logsource = $log_sources[$i] -split '%'
  if ($logsource.Length -ne 2) { continue }

  $log_source = $($logsource[0])
  $log_sourcetype = $($logsource[1])

  if ($log_sourcetype -eq 'WinEventLog') {
    #Get-Variable -Name $log_source -ValueOnly -ErrorAction 'Ignore' >> $inputconfTemp
    echo "[WinEventLog://$log_source]" >> $inputconfTemp
    echo "$eventlog_options" >> $inputconfTemp
    echo "`r`n" >> $inputconfTemp
  }
  else {
    echo "[monitor://$log_source]" >> $inputconfTemp
    echo "sourcetype = $log_sourcetype" >> $inputconfTemp
    echo "disabled = false" >> $inputconfTemp
    if ($log_sourcetype -eq 'collectd' -Or $log_sourcetype -eq 'uf') {
      echo "index = _internal" >> $inputconfTemp
    }
    echo "`r`n" >> $inputconfTemp
  }
}

if (Test-Path -Path $splunkpath"inputs.old" ) {Remove-Item -Path $splunkpath"inputs.old"}
if (Test-Path -Path $splunkpath"inputs.conf" ) {Rename-Item -Path $splunkpath"inputs.conf" -NewName $splunkpath"inputs.old"}
if (Test-Path -Path $splunkpath"inputs.conf" ) {Remove-Item -Path $splunkpath"inputs.conf"} 

Rename-Item -Path $inputconfTemp -NewName $splunkpath"inputs.conf"

TO Roll-out with Forwarder Manager:
Create the new Applications on the Forwarder Manager
Name: ManagedInfrastructure

etc/deployment-apps/ManagedInfrastructure/bin/install.path
etc/deployment-apps/ManagedInfrastructure/bin/install.ps1
etc/deployment-apps/ManagedInfrastructure/local/inputs.conf

Name: Restart_Only
etc/deployment-apps/Restart_Only

Set Ownership of new files

chown -R splunk:splunk /opt/splunk/etc/deployment-apps/ManagedInfrastructure/
chown -R splunk:splunk /opt/splunk/etc/deployment-apps/Restart_Only

install.path: C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -ExecutionPolicy RemoteSigned -Command "& 'C:\Program Files\SplunkUniversalForwarder\etc\apps\ManagedInfrastructure\bin\install.ps1'

inputs.conf

[script://.\bin\install.path]
interval = -1

Push out with the Manager, then Push out Restart_Only app.

[sloshburch]

This is a fast developing product so keep in mind that a lot of this may change the effectiveness of this script over time. Just saying in case someone finds our time capsule in the future.

[qhmassc]

I didn't see any indexing error like " metric event not indexed"., but I do see parsing error:
01-07-2019 17:18:04.592 -0500 ERROR HttpInputDataHandler - Parsing error : Server is busy

[dagarwal_splunk · Accepted solution]

You need to setup inputs.conf (add all metrics and logs data to collect) and outputs.conf (send data to SAI instance) on existing Splunk Universal Forwarders.

Here is useful link:
http://docs.splunk.com/Documentation/InfraApp/1.2.2/Admin/ManualInstallWindowsUF

Something similar to this :
https://answers.splunk.com/answers/699711/can-you-help-me-use-the-splunk-app-for-infrastruct.html#an...

[omprakash9998]

Can we install the Splunk App for Infrastructure on the same search head we have been using to search our data. If yes, so all that we have to change is the index to which the data is routing.?

[dagarwal_splunk]

yes, SAI will be installed in the search head.

• Index is set as "em_metrics" in UF's inputs.conf for "perfmon:*" metrics data as mentioned in the above links.
• Just make sure you have Add-on for infra installed in your indexers. https://splunkbase.splunk.com/app/4217/
• See the docs for more info http://docs.splunk.com/Documentation/InfraApp/1.2.2/Install/DistributedDeployment

[omprakash9998]

It Worked.

Thank you

[qhmassc]

I have similar environment, If I go to Investigate > Entiries, I can see the Entity Name, if I click Entity Name, I can see overview with number of events, but no results found for CPU, Disk Memory and Network. Please help.

[dagarwal_splunk]

• What version of SAI(Splunk App for Infra) are you using?
• Do you have Splunk Add-on for Windows installed on the same instance as SAI? If yes, that may be the problem.

[florianduhme]

Just found out that I had both add-ons "Splunk Add-on for Microsoft Windows" and "Splunk Add-on for Infrastructure" installed on our Heavy Forwarder.
After disabling the "Splunk Add-on for Microsoft Windows" the metrics and logs are now forwarded to the Splunk app for infrastructure.
Turns out that you can not/should not install both on the same forwarder.

[sloshburch]

If it helps: What is the best way to migrate Windows performance monitoring from event-based to metrics-based dat...

[qhmassc]

SAI version: 1.2.1, I also have Splunk Add-on for Infrastructure 1.2.1 installed, but no Splunk Add-on for Windows installed.

[dagarwal_splunk]

@qhmassc Try adding "entity_type::Windows_Host" to your _meta fields in "inputs.conf"

It should look like:

[perfmon://CPU Load]
counters = % C1 Time;% C2 Time;% Idle Time;% Processor Time;% User Time;% Privileged Time;% Reserved Time;% Interrupt Time
instances = *
interval = 30
object = Processor
index = em_metrics
_meta = os::"Microsoft Windows" entity_type::Windows_Host

Similarly add other perfmon data you want to add using the sample inputs.conf file.

[sloshburch]

It seems the _meta = entity_type::Windows_Host is a lot more critical than the documentation explains. I've reached out to the product team about it to see if we can better explain that requirement.

[qhmassc]

I have Splunk App for Infrastructure installed and configured, it works for Windows agent---Thanks for your help, but I cannot make it for Linux servers.
I run the install script, I can start both collectd and splunk agent services, collectd.log file is located under /etc/collectd/. But I cannot see Linux servers under Entities with Splunk App for Infrastructure. I can see other log files if I use searching and reporting app.

Please help.

[dagarwal_splunk]

Are there any errors in /etc/collectd/collectd.log file ?

Something like "curl failed" ?

[qhmassc]

[2019-01-04 10:43:01] [info] Initialization complete, entering read-loop.
[2019-01-04 11:28:17] [info] Exiting normally.
[2019-01-04 11:28:17] [info] collectd: Stopping 5 read threads.
[2019-01-04 11:28:17] [info] collectd: Stopping 5 write threads.
[2019-01-04 11:28:17] [info] Initialization complete, entering read-loop.
[2019-01-04 11:57:51] [info] Exiting normally.
[2019-01-04 11:57:51] [info] collectd: Stopping 5 read threads.
[2019-01-04 11:57:51] [info] collectd: Stopping 5 write threads.
[2019-01-04 11:57:51] [info] Initialization complete, entering read-loop.
[2019-01-04 12:25:35] [info] Exiting normally.
[2019-01-04 12:25:35] [info] collectd: Stopping 5 read threads.
[2019-01-04 12:25:35] [info] collectd: Stopping 5 write threads.
[2019-01-04 12:25:35] [info] Initialization complete, entering read-loop.
[2019-01-04 12:29:45] [info] Exiting normally.
[2019-01-04 12:29:45] [info] collectd: Stopping 5 read threads.
[2019-01-04 12:29:45] [info] collectd: Stopping 5 write threads.
[2019-01-04 12:30:49] [info] Initialization complete, entering read-loop.

[dagarwal_splunk]

Your token should look like : "xxx2fb15-5c36-489e-b624-xyzgdshg" .

Not the name of token?

[qhmassc]

my token looks like : "xxx2fb15-5c36-489e-b624-xyzgdshg", Not the name of token.

Dimension "entity_type:Linux_Host" was comment out.

[dagarwal_splunk]

Let's check /opt/splunk/var/log/splunk/splunkd.log for any indexing error like " metric event not indexed".

Also, can you try this curl from Linux host as well mentioned in doc: (https://docs.splunk.com/Documentation/Splunk/7.2.3/Data/Sendmetricstoametricsindex)

curl -k https://localhost:8088/services/collector \
-H "Authorization: Splunk b0221cd8-c4b4-465a-9a3c-273e3a75aa29" \
-d '{"time": 1486683865.000,"source":"disk","host":"host_99","fields":{"region":"us-west-1","datacenter":"us-west-1a","rack":"63","os":"Ubuntu16.10","arch":"x64","team":"LON","service":"6","service_version":"0","service_environment":"test","path":"/dev/sda1","fstype":"ext3","_value":1099511627776,"metric_name":"total"}}'

Replace token with yours and localhost with your SAI instance. See, if you get success.