Hi,
We have a Splunk environment with universal forwarders already installed on our Windows servers. We want to try the Splunk App for infrastructure. Can we use the existing Universal Forwarders to use the Splunk App for Infrastructure without having to running the script and reinstalling on all of our servers.
Thanks.
You need to setup inputs.conf (add all metrics and logs data to collect) and outputs.conf (send data to SAI instance) on existing Splunk Universal Forwarders.
Here is useful link:
http://docs.splunk.com/Documentation/InfraApp/1.2.2/Admin/ManualInstallWindowsUF
Something similar to this :
https://answers.splunk.com/answers/699711/can-you-help-me-use-the-splunk-app-for-infrastruct.html#an...
Here is a script that you can use. I use it to roll this out using Forwarder Management - but you can just run it on each host if you don't have too many. I named the add-on "ManagedInfrastructure" on the Univ Forwarders, but you can keep the original name just change that line in the script. If you want to use your Forwarder Manager, I have included that after the script.
install.ps1
$splunkpath = "C:\Program Files\SplunkUniversalForwarder\etc\apps\ManagedInfrastructure\local\" $inputconfTemp = $splunkpath + "inputs.temp"
$metrics_param = "logical_disk,physical_disk,cpu,memory,network,system,process"
$log_sources_param = "application%WinEventLog,system%WinEventLog,security%WinEventLog"
$cpu_instance_type = '_Total'
$metrics_index = "em_metrics"
if (Test-Path -Path $inputconfTemp ) {Remove-Item -Path $inputconfTemp}
$m_cpu =`
"[perfmon://CPU]`r`n" `
+ "counters = % C1 Time;% C2 Time;% Idle Time;% Processor Time;% User Time;% Privileged Time;% Reserved Time;% Interrupt Time`r`n" `
+ "instances = $cpu_instance_type`r`n" `
+ "interval = 30`r`n" `
+ "object = Processor`r`n" `
+ "mode = single`r`n" `
+ "useEnglishOnly = true`r`n" `
+ "index = $metrics_index"
$m_memory =`
"[perfmon://Memory]`r`n" `
+ "counters = Cache Bytes;% Committed Bytes In Use;Page Reads/sec;Pages Input/sec;Pages Output/sec;Committed Bytes;Available
Bytes`r`n"`
+ "interval = 30`r`n" `
+ "object = Memory`r`n" `
+ "mode = single`r`n" `
+ "useEnglishOnly = true`r`n" `
+ "index = $metrics_index"
$m_physical_disk =`
"[perfmon://PhysicalDisk]`r`n" `
+ "counters = % Disk Read Time;% Disk Write Time`r`n" `
+ "instances = *`r`n" `
+ "interval = 30`r`n" `
+ "object = PhysicalDisk`r`n" `
+ "mode = single`r`n" `
+ "useEnglishOnly = true`r`n" `
+ "index = $metrics_index"
$m_logical_disk =`
"[perfmon://LogicalDisk]`r`n" `
+ "counters = Free Megabytes;% Free Space`r`n" `
+ "instances = *`r`n" `
+ "interval = 30`r`n" `
+ "object = LogicalDisk`r`n" `
+ "mode = single`r`n" `
+ "useEnglishOnly = true`r`n" `
+ "index = $metrics_index"
$m_network =`
"[perfmon://Network]`r`n" `
+ "counters = Bytes Received/sec;Bytes Sent/sec;Packets Received/sec;Packets Sent/sec;Packets Received Errors;Packets Outbound
Errors`r`n" `
+ "instances = *`r`n" `
+ "interval = 30`r`n" `
+ "mode = single`r`n" `
+ "object = Network Interface`r`n" `
+ "useEnglishOnly = true`r`n" `
+ "index = $metrics_index"
$m_system =`
"[perfmon://System]`r`n" `
+ "counters = Processor Queue Length;Threads`r`n" `
+ "instances = *`r`n" `
+ "interval = 30`r`n" `
+ "object = System`r`n" `
+ "mode = single`r`n" `
+ "useEnglishOnly = true`r`n" `
+ "index = $metrics_index"
$m_process =`
"[perfmon://Process]`r`n" `
+ "counters = % Processor Time;% User Time;% Privileged Time`r`n" `
+ "instances = *`r`n" `
+ "interval = 30`r`n" `
+ "object = Process`r`n" `
+ "mode = single`r`n" `
+ "useEnglishOnly = true`r`n" `
+ "index = $metrics_index"
# Inputs.conf options for the supported Windows EventLogs
$eventlog_options =`
"checkpointInterval = 10`r`n" `
+ "current_only = 1`r`n" `
+ "disabled = 0`r`n" `
+ "start_from = oldest"
# extract os & ip info
# > WIN 7
#$os_info = Get-CimInstance Win32_OperatingSystem | Select-Object Caption, Version
# =WIN 7
$os_info = Get-WmiObject Win32_OperatingSystem | Select-Object Caption, Version
$ip_info = Test-Connection -ComputerName $env:computername -count 1 | Select-Object IPV4Address
# Add OS & ip info as dimension
# If these dimensions are to be updated, user has to rerun this script
$dims = $dimensions + " os::" + "`"" + $os_info.Caption + "`""
$dims = $dims + " os_version::" + $os_info.Version
$dims = $dims + " ip::" + "`"" + $ip_info.IPV4Address.IPAddressToString + "`""
$dims = $dims + ' entity_type::Windows_Host'
$metrics = $metrics_param -split ','
$log_sources = $log_sources_param -split ','
echo "# *** Configure Metrics Logs collected ***" > $inputconfTemp
For ($i=0; $i -lt $metrics.Length; $i++) {
$m_name = "m_" + $metrics[$i]
Get-Variable -Name $m_name -ValueOnly -ErrorAction 'Ignore' >> $inputconfTemp
# Add dimensions
echo "_meta = $dims" >> $inputconfTemp
echo "`n" >> $inputconfTemp
}
For ($i=0; $i -lt $log_sources.Length; $i++) {
if([string]::IsNullOrEmpty($log_sources[$i])) { continue }
# split log source into source and sourcetype
$logsource = $log_sources[$i] -split '%'
if ($logsource.Length -ne 2) { continue }
$log_source = $($logsource[0])
$log_sourcetype = $($logsource[1])
if ($log_sourcetype -eq 'WinEventLog') {
#Get-Variable -Name $log_source -ValueOnly -ErrorAction 'Ignore' >> $inputconfTemp
echo "[WinEventLog://$log_source]" >> $inputconfTemp
echo "$eventlog_options" >> $inputconfTemp
echo "`r`n" >> $inputconfTemp
}
else {
echo "[monitor://$log_source]" >> $inputconfTemp
echo "sourcetype = $log_sourcetype" >> $inputconfTemp
echo "disabled = false" >> $inputconfTemp
if ($log_sourcetype -eq 'collectd' -Or $log_sourcetype -eq 'uf') {
echo "index = _internal" >> $inputconfTemp
}
echo "`r`n" >> $inputconfTemp
}
}
if (Test-Path -Path $splunkpath"inputs.old" ) {Remove-Item -Path $splunkpath"inputs.old"}
if (Test-Path -Path $splunkpath"inputs.conf" ) {Rename-Item -Path $splunkpath"inputs.conf" -NewName $splunkpath"inputs.old"}
if (Test-Path -Path $splunkpath"inputs.conf" ) {Remove-Item -Path $splunkpath"inputs.conf"}
Rename-Item -Path $inputconfTemp -NewName $splunkpath"inputs.conf"
TO Roll-out with Forwarder Manager:
Create the new Applications on the Forwarder Manager
Name: ManagedInfrastructure
etc/deployment-apps/ManagedInfrastructure/bin/install.path
etc/deployment-apps/ManagedInfrastructure/bin/install.ps1
etc/deployment-apps/ManagedInfrastructure/local/inputs.conf
Name: Restart_Only
etc/deployment-apps/Restart_Only
Set Ownership of new files
chown -R splunk:splunk /opt/splunk/etc/deployment-apps/ManagedInfrastructure/
chown -R splunk:splunk /opt/splunk/etc/deployment-apps/Restart_Only
install.path: C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -ExecutionPolicy RemoteSigned -Command "& 'C:\Program Files\SplunkUniversalForwarder\etc\apps\ManagedInfrastructure\bin\install.ps1'
inputs.conf
[script://.\bin\install.path]
interval = -1
Push out with the Manager, then Push out Restart_Only app.
This is a fast developing product so keep in mind that a lot of this may change the effectiveness of this script over time. Just saying in case someone finds our time capsule in the future.
I didn't see any indexing error like " metric event not indexed"., but I do see parsing error:
01-07-2019 17:18:04.592 -0500 ERROR HttpInputDataHandler - Parsing error : Server is busy
You need to setup inputs.conf (add all metrics and logs data to collect) and outputs.conf (send data to SAI instance) on existing Splunk Universal Forwarders.
Here is useful link:
http://docs.splunk.com/Documentation/InfraApp/1.2.2/Admin/ManualInstallWindowsUF
Something similar to this :
https://answers.splunk.com/answers/699711/can-you-help-me-use-the-splunk-app-for-infrastruct.html#an...
Can we install the Splunk App for Infrastructure on the same search head we have been using to search our data. If yes, so all that we have to change is the index to which the data is routing.?
yes, SAI will be installed in the search head.
It Worked.
Thank you
I have similar environment, If I go to Investigate > Entiries, I can see the Entity Name, if I click Entity Name, I can see overview with number of events, but no results found for CPU, Disk Memory and Network. Please help.
Just found out that I had both add-ons "Splunk Add-on for Microsoft Windows" and "Splunk Add-on for Infrastructure" installed on our Heavy Forwarder.
After disabling the "Splunk Add-on for Microsoft Windows" the metrics and logs are now forwarded to the Splunk app for infrastructure.
Turns out that you can not/should not install both on the same forwarder.
SAI version: 1.2.1, I also have Splunk Add-on for Infrastructure 1.2.1 installed, but no Splunk Add-on for Windows installed.
@qhmassc Try adding "entity_type::Windows_Host" to your _meta fields in "inputs.conf"
It should look like:
[perfmon://CPU Load]
counters = % C1 Time;% C2 Time;% Idle Time;% Processor Time;% User Time;% Privileged Time;% Reserved Time;% Interrupt Time
instances = *
interval = 30
object = Processor
index = em_metrics
_meta = os::"Microsoft Windows" entity_type::Windows_Host
Similarly add other perfmon data you want to add using the sample inputs.conf file.
It seems the _meta = entity_type::Windows_Host
is a lot more critical than the documentation explains. I've reached out to the product team about it to see if we can better explain that requirement.
I have Splunk App for Infrastructure installed and configured, it works for Windows agent---Thanks for your help, but I cannot make it for Linux servers.
I run the install script, I can start both collectd and splunk agent services, collectd.log file is located under /etc/collectd/. But I cannot see Linux servers under Entities with Splunk App for Infrastructure. I can see other log files if I use searching and reporting app.
Please help.
Are there any errors in /etc/collectd/collectd.log file ?
Something like "curl failed" ?
[2019-01-04 10:43:01] [info] Initialization complete, entering read-loop.
[2019-01-04 11:28:17] [info] Exiting normally.
[2019-01-04 11:28:17] [info] collectd: Stopping 5 read threads.
[2019-01-04 11:28:17] [info] collectd: Stopping 5 write threads.
[2019-01-04 11:28:17] [info] Initialization complete, entering read-loop.
[2019-01-04 11:57:51] [info] Exiting normally.
[2019-01-04 11:57:51] [info] collectd: Stopping 5 read threads.
[2019-01-04 11:57:51] [info] collectd: Stopping 5 write threads.
[2019-01-04 11:57:51] [info] Initialization complete, entering read-loop.
[2019-01-04 12:25:35] [info] Exiting normally.
[2019-01-04 12:25:35] [info] collectd: Stopping 5 read threads.
[2019-01-04 12:25:35] [info] collectd: Stopping 5 write threads.
[2019-01-04 12:25:35] [info] Initialization complete, entering read-loop.
[2019-01-04 12:29:45] [info] Exiting normally.
[2019-01-04 12:29:45] [info] collectd: Stopping 5 read threads.
[2019-01-04 12:29:45] [info] collectd: Stopping 5 write threads.
[2019-01-04 12:30:49] [info] Initialization complete, entering read-loop.
Your token should look like : "xxx2fb15-5c36-489e-b624-xyzgdshg" .
Not the name of token?
my token looks like : "xxx2fb15-5c36-489e-b624-xyzgdshg", Not the name of token.
Let's check /opt/splunk/var/log/splunk/splunkd.log for any indexing error like " metric event not indexed".
Also, can you try this curl from Linux host as well mentioned in doc: (https://docs.splunk.com/Documentation/Splunk/7.2.3/Data/Sendmetricstoametricsindex)
curl -k https://localhost:8088/services/collector \
-H "Authorization: Splunk b0221cd8-c4b4-465a-9a3c-273e3a75aa29" \
-d '{"time": 1486683865.000,"source":"disk","host":"host_99","fields":{"region":"us-west-1","datacenter":"us-west-1a","rack":"63","os":"Ubuntu16.10","arch":"x64","team":"LON","service":"6","service_version":"0","service_environment":"test","path":"/dev/sda1","fstype":"ext3","_value":1099511627776,"metric_name":"total"}}'
Replace token with yours and localhost with your SAI instance. See, if you get success.