All Apps and Add-ons

In the Splunk App for Infrastructure, can you use existing universal forwarders without running the script and reinstalling on all of our servers?

omprakash9998
Path Finder

Hi,

We have a Splunk environment with universal forwarders already installed on our Windows servers. We want to try the Splunk App for infrastructure. Can we use the existing Universal Forwarders to use the Splunk App for Infrastructure without having to running the script and reinstalling on all of our servers.

Thanks.

1 Solution

dagarwal_splunk
Splunk Employee
Splunk Employee

You need to setup inputs.conf (add all metrics and logs data to collect) and outputs.conf (send data to SAI instance) on existing Splunk Universal Forwarders.

Here is useful link:
http://docs.splunk.com/Documentation/InfraApp/1.2.2/Admin/ManualInstallWindowsUF

Something similar to this :
https://answers.splunk.com/answers/699711/can-you-help-me-use-the-splunk-app-for-infrastruct.html#an...

View solution in original post

myfriendhenry
Explorer

Here is a script that you can use. I use it to roll this out using Forwarder Management - but you can just run it on each host if you don't have too many. I named the add-on "ManagedInfrastructure" on the Univ Forwarders, but you can keep the original name just change that line in the script. If you want to use your Forwarder Manager, I have included that after the script.

install.ps1

$splunkpath = "C:\Program Files\SplunkUniversalForwarder\etc\apps\ManagedInfrastructure\local\" $inputconfTemp = $splunkpath + "inputs.temp"

  $metrics_param = "logical_disk,physical_disk,cpu,memory,network,system,process"
  $log_sources_param = "application%WinEventLog,system%WinEventLog,security%WinEventLog"
  $cpu_instance_type = '_Total'
  $metrics_index = "em_metrics"

if (Test-Path -Path $inputconfTemp ) {Remove-Item -Path $inputconfTemp}


$m_cpu =`
"[perfmon://CPU]`r`n" `
+ "counters = % C1 Time;% C2 Time;% Idle Time;% Processor Time;% User Time;% Privileged Time;% Reserved Time;% Interrupt Time`r`n" `
+ "instances = $cpu_instance_type`r`n" `
+ "interval = 30`r`n" `
+ "object = Processor`r`n" `
+ "mode = single`r`n" `
+ "useEnglishOnly = true`r`n" `
+ "index = $metrics_index"

$m_memory =`
"[perfmon://Memory]`r`n" `
+ "counters = Cache Bytes;% Committed Bytes In Use;Page Reads/sec;Pages Input/sec;Pages Output/sec;Committed Bytes;Available  
Bytes`r`n"`
+ "interval = 30`r`n" `
+ "object = Memory`r`n" `
+ "mode = single`r`n" `
+ "useEnglishOnly = true`r`n" `
+ "index = $metrics_index"

$m_physical_disk =`
"[perfmon://PhysicalDisk]`r`n" `
+ "counters = % Disk Read Time;% Disk Write Time`r`n" `
+ "instances = *`r`n" `
+ "interval = 30`r`n" `
+ "object = PhysicalDisk`r`n" `
+ "mode = single`r`n" `
+ "useEnglishOnly = true`r`n" `
+ "index = $metrics_index"

$m_logical_disk =`
"[perfmon://LogicalDisk]`r`n" `
+ "counters = Free Megabytes;% Free Space`r`n" `
+ "instances = *`r`n" `
+ "interval = 30`r`n" `
+ "object = LogicalDisk`r`n" `
+ "mode = single`r`n" `
+ "useEnglishOnly = true`r`n" `
+ "index = $metrics_index"

$m_network =`
"[perfmon://Network]`r`n" `
+ "counters = Bytes Received/sec;Bytes Sent/sec;Packets Received/sec;Packets Sent/sec;Packets Received Errors;Packets Outbound  
Errors`r`n" `
+ "instances = *`r`n" `
+ "interval = 30`r`n" `
+ "mode = single`r`n" `
+ "object = Network Interface`r`n" `
+ "useEnglishOnly = true`r`n" `
+ "index = $metrics_index"

$m_system =`
"[perfmon://System]`r`n" `
+ "counters = Processor Queue Length;Threads`r`n" `
+ "instances = *`r`n" `
+ "interval = 30`r`n" `
+ "object = System`r`n" `
+ "mode = single`r`n" `
+ "useEnglishOnly = true`r`n" `
+ "index = $metrics_index"

$m_process =`
"[perfmon://Process]`r`n" `
+ "counters = % Processor Time;% User Time;% Privileged Time`r`n" `
+ "instances = *`r`n" `
+ "interval = 30`r`n" `
+ "object = Process`r`n" `
+ "mode = single`r`n" `
+ "useEnglishOnly = true`r`n" `
+ "index = $metrics_index"

# Inputs.conf options for the supported Windows EventLogs
$eventlog_options =`
"checkpointInterval = 10`r`n" `
+ "current_only = 1`r`n" `
+ "disabled = 0`r`n" `
+ "start_from = oldest"  

# extract os & ip info
# > WIN 7
#$os_info = Get-CimInstance Win32_OperatingSystem | Select-Object Caption, Version

# =WIN 7
$os_info = Get-WmiObject Win32_OperatingSystem | Select-Object Caption, Version
$ip_info = Test-Connection -ComputerName $env:computername -count 1 | Select-Object IPV4Address

# Add OS & ip info as dimension
# If these dimensions are to be updated, user has to rerun this script
$dims = $dimensions + " os::" + "`"" + $os_info.Caption + "`""
$dims = $dims + " os_version::" + $os_info.Version
$dims = $dims + " ip::" + "`"" + $ip_info.IPV4Address.IPAddressToString + "`""
$dims = $dims + ' entity_type::Windows_Host'

$metrics = $metrics_param -split ','
$log_sources = $log_sources_param -split ','

echo "# *** Configure Metrics Logs collected ***" > $inputconfTemp


For ($i=0; $i -lt $metrics.Length; $i++) {
  $m_name = "m_" + $metrics[$i]
  Get-Variable -Name $m_name -ValueOnly -ErrorAction 'Ignore' >> $inputconfTemp
  # Add dimensions
  echo "_meta = $dims" >> $inputconfTemp
  echo "`n" >> $inputconfTemp
}


For ($i=0; $i -lt $log_sources.Length; $i++) {
  if([string]::IsNullOrEmpty($log_sources[$i])) { continue }
  # split log source into source and sourcetype
  $logsource = $log_sources[$i] -split '%'
  if ($logsource.Length -ne 2) { continue }

  $log_source = $($logsource[0])
  $log_sourcetype = $($logsource[1])

  if ($log_sourcetype -eq 'WinEventLog') {
    #Get-Variable -Name $log_source -ValueOnly -ErrorAction 'Ignore' >> $inputconfTemp
    echo "[WinEventLog://$log_source]" >> $inputconfTemp
    echo "$eventlog_options" >> $inputconfTemp
    echo "`r`n" >> $inputconfTemp
  }
  else {
    echo "[monitor://$log_source]" >> $inputconfTemp
    echo "sourcetype = $log_sourcetype" >> $inputconfTemp
    echo "disabled = false" >> $inputconfTemp
    if ($log_sourcetype -eq 'collectd' -Or $log_sourcetype -eq 'uf') {
      echo "index = _internal" >> $inputconfTemp
    }
    echo "`r`n" >> $inputconfTemp
  }
}

if (Test-Path -Path $splunkpath"inputs.old" ) {Remove-Item -Path $splunkpath"inputs.old"}
if (Test-Path -Path $splunkpath"inputs.conf" ) {Rename-Item -Path $splunkpath"inputs.conf" -NewName $splunkpath"inputs.old"}
if (Test-Path -Path $splunkpath"inputs.conf" ) {Remove-Item -Path $splunkpath"inputs.conf"} 

Rename-Item -Path $inputconfTemp -NewName $splunkpath"inputs.conf"

TO Roll-out with Forwarder Manager:
Create the new Applications on the Forwarder Manager
Name: ManagedInfrastructure

etc/deployment-apps/ManagedInfrastructure/bin/install.path
etc/deployment-apps/ManagedInfrastructure/bin/install.ps1
etc/deployment-apps/ManagedInfrastructure/local/inputs.conf

Name: Restart_Only
etc/deployment-apps/Restart_Only

Set Ownership of new files

chown -R splunk:splunk /opt/splunk/etc/deployment-apps/ManagedInfrastructure/
chown -R splunk:splunk /opt/splunk/etc/deployment-apps/Restart_Only

install.path: C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -ExecutionPolicy RemoteSigned -Command "& 'C:\Program Files\SplunkUniversalForwarder\etc\apps\ManagedInfrastructure\bin\install.ps1'

inputs.conf

[script://.\bin\install.path]
interval = -1

Push out with the Manager, then Push out Restart_Only app.

0 Karma

sloshburch
Ultra Champion

This is a fast developing product so keep in mind that a lot of this may change the effectiveness of this script over time. Just saying in case someone finds our time capsule in the future.

0 Karma

qhmassc
Explorer

I didn't see any indexing error like " metric event not indexed"., but I do see parsing error:
01-07-2019 17:18:04.592 -0500 ERROR HttpInputDataHandler - Parsing error : Server is busy

0 Karma

dagarwal_splunk
Splunk Employee
Splunk Employee

You need to setup inputs.conf (add all metrics and logs data to collect) and outputs.conf (send data to SAI instance) on existing Splunk Universal Forwarders.

Here is useful link:
http://docs.splunk.com/Documentation/InfraApp/1.2.2/Admin/ManualInstallWindowsUF

Something similar to this :
https://answers.splunk.com/answers/699711/can-you-help-me-use-the-splunk-app-for-infrastruct.html#an...

View solution in original post

omprakash9998
Path Finder

Can we install the Splunk App for Infrastructure on the same search head we have been using to search our data. If yes, so all that we have to change is the index to which the data is routing.?

0 Karma

dagarwal_splunk
Splunk Employee
Splunk Employee

yes, SAI will be installed in the search head.

0 Karma

omprakash9998
Path Finder

It Worked.

Thank you

0 Karma

qhmassc
Explorer

I have similar environment, If I go to Investigate > Entiries, I can see the Entity Name, if I click Entity Name, I can see overview with number of events, but no results found for CPU, Disk Memory and Network. Please help.

0 Karma

dagarwal_splunk
Splunk Employee
Splunk Employee
  • What version of SAI(Splunk App for Infra) are you using?
  • Do you have Splunk Add-on for Windows installed on the same instance as SAI? If yes, that may be the problem.

florianduhme
Path Finder

Just found out that I had both add-ons "Splunk Add-on for Microsoft Windows" and "Splunk Add-on for Infrastructure" installed on our Heavy Forwarder.
After disabling the "Splunk Add-on for Microsoft Windows" the metrics and logs are now forwarded to the Splunk app for infrastructure.
Turns out that you can not/should not install both on the same forwarder.

0 Karma

sloshburch
Ultra Champion
0 Karma

qhmassc
Explorer

SAI version: 1.2.1, I also have Splunk Add-on for Infrastructure 1.2.1 installed, but no Splunk Add-on for Windows installed.

0 Karma

dagarwal_splunk
Splunk Employee
Splunk Employee

@qhmassc Try adding "entity_type::Windows_Host" to your _meta fields in "inputs.conf"

It should look like:

[perfmon://CPU Load]
counters = % C1 Time;% C2 Time;% Idle Time;% Processor Time;% User Time;% Privileged Time;% Reserved Time;% Interrupt Time
instances = *
interval = 30
object = Processor
index = em_metrics
_meta = os::"Microsoft Windows" entity_type::Windows_Host

Similarly add other perfmon data you want to add using the sample inputs.conf file.

0 Karma

sloshburch
Ultra Champion

It seems the _meta = entity_type::Windows_Host is a lot more critical than the documentation explains. I've reached out to the product team about it to see if we can better explain that requirement.

0 Karma

qhmassc
Explorer

I have Splunk App for Infrastructure installed and configured, it works for Windows agent---Thanks for your help, but I cannot make it for Linux servers.
I run the install script, I can start both collectd and splunk agent services, collectd.log file is located under /etc/collectd/. But I cannot see Linux servers under Entities with Splunk App for Infrastructure. I can see other log files if I use searching and reporting app.

Please help.

0 Karma

dagarwal_splunk
Splunk Employee
Splunk Employee

Are there any errors in /etc/collectd/collectd.log file ?

Something like "curl failed" ?

0 Karma

qhmassc
Explorer

[2019-01-04 10:43:01] [info] Initialization complete, entering read-loop.
[2019-01-04 11:28:17] [info] Exiting normally.
[2019-01-04 11:28:17] [info] collectd: Stopping 5 read threads.
[2019-01-04 11:28:17] [info] collectd: Stopping 5 write threads.
[2019-01-04 11:28:17] [info] Initialization complete, entering read-loop.
[2019-01-04 11:57:51] [info] Exiting normally.
[2019-01-04 11:57:51] [info] collectd: Stopping 5 read threads.
[2019-01-04 11:57:51] [info] collectd: Stopping 5 write threads.
[2019-01-04 11:57:51] [info] Initialization complete, entering read-loop.
[2019-01-04 12:25:35] [info] Exiting normally.
[2019-01-04 12:25:35] [info] collectd: Stopping 5 read threads.
[2019-01-04 12:25:35] [info] collectd: Stopping 5 write threads.
[2019-01-04 12:25:35] [info] Initialization complete, entering read-loop.
[2019-01-04 12:29:45] [info] Exiting normally.
[2019-01-04 12:29:45] [info] collectd: Stopping 5 read threads.
[2019-01-04 12:29:45] [info] collectd: Stopping 5 write threads.
[2019-01-04 12:30:49] [info] Initialization complete, entering read-loop.

0 Karma

dagarwal_splunk
Splunk Employee
Splunk Employee

Your token should look like : "xxx2fb15-5c36-489e-b624-xyzgdshg" .

Not the name of token?

0 Karma

qhmassc
Explorer

my token looks like : "xxx2fb15-5c36-489e-b624-xyzgdshg", Not the name of token.

Dimension "entity_type:Linux_Host" was comment out.

0 Karma

dagarwal_splunk
Splunk Employee
Splunk Employee

Let's check /opt/splunk/var/log/splunk/splunkd.log for any indexing error like " metric event not indexed".

Also, can you try this curl from Linux host as well mentioned in doc: (https://docs.splunk.com/Documentation/Splunk/7.2.3/Data/Sendmetricstoametricsindex)

curl -k https://localhost:8088/services/collector \
-H "Authorization: Splunk b0221cd8-c4b4-465a-9a3c-273e3a75aa29" \
-d '{"time": 1486683865.000,"source":"disk","host":"host_99","fields":{"region":"us-west-1","datacenter":"us-west-1a","rack":"63","os":"Ubuntu16.10","arch":"x64","team":"LON","service":"6","service_version":"0","service_environment":"test","path":"/dev/sda1","fstype":"ext3","_value":1099511627776,"metric_name":"total"}}'

Replace token with yours and localhost with your SAI instance. See, if you get success.

0 Karma
.conf21 CFS Extended through 5/20!

Don't miss your chance
to share your Splunk
wisdom in-person or
virtually at .conf21!

Call for Speakers has
been extended through
Thursday, 5/20!