OK, got BOTH SH's sending metrics to ONE SH, the other SH only gets it's OWN metrics - thinking network issue. Unable to get ANY metrics out of the MASTER however. ... View more

my boss advised the following: So you will need to install the Add-on and the inputs.conf file onto the Splunk Enterprise server itself under /opt/splunk/etc/apps/Add-on/local/inputs.conf. which should match the inputs.conf that you have deployed out to the forwarders under /opt/splunkforwarder/etc/apps/Add-on/local/inputs.conf. And the data should start being collected. If the Splunk Enterprise server is not the indexer, make sure that you have your outputs.conf configured on the server under /opt/splunk/etc/system/local/outputs.conf which should match your forwarders under /opt/splunkforwarder/etc/system/local/outputs.conf, unless you specify it under another add-on. ... View more

I understand the collectd part, np. I use the manual method and open a udp port as in "https://docs.splunk.com/Documentation/InfraApp/1.3.0/Admin/WriteCollectdToUF" and it works perfectly for my universal forwarders. So I got all of that. My confusion is WHERE to put INPUTS.CONF on the SEARCHHEADS and INDEXERS to monitor THEM with the app. ... View more

Here is a script that you can use. I use it to roll this out using Forwarder Management - but you can just run it on each host if you don't have too many. I named the add-on "ManagedInfrastructure" on the Univ Forwarders, but you can keep the original name just change that line in the script. If you want to use your Forwarder Manager, I have included that after the script. install.ps1 $splunkpath = "C:\Program Files\SplunkUniversalForwarder\etc\apps\ManagedInfrastructure\local\" $inputconfTemp = $splunkpath + "inputs.temp" $metrics_param = "logical_disk,physical_disk,cpu,memory,network,system,process" $log_sources_param = "application%WinEventLog,system%WinEventLog,security%WinEventLog" $cpu_instance_type = '_Total' $metrics_index = "em_metrics" if (Test-Path -Path $inputconfTemp ) {Remove-Item -Path $inputconfTemp} $m_cpu =` "[perfmon://CPU]`r`n" ` + "counters = % C1 Time;% C2 Time;% Idle Time;% Processor Time;% User Time;% Privileged Time;% Reserved Time;% Interrupt Time`r`n" ` + "instances = $cpu_instance_type`r`n" ` + "interval = 30`r`n" ` + "object = Processor`r`n" ` + "mode = single`r`n" ` + "useEnglishOnly = true`r`n" ` + "index = $metrics_index" $m_memory =` "[perfmon://Memory]`r`n" ` + "counters = Cache Bytes;% Committed Bytes In Use;Page Reads/sec;Pages Input/sec;Pages Output/sec;Committed Bytes;Available Bytes`r`n"` + "interval = 30`r`n" ` + "object = Memory`r`n" ` + "mode = single`r`n" ` + "useEnglishOnly = true`r`n" ` + "index = $metrics_index" $m_physical_disk =` "[perfmon://PhysicalDisk]`r`n" ` + "counters = % Disk Read Time;% Disk Write Time`r`n" ` + "instances = *`r`n" ` + "interval = 30`r`n" ` + "object = PhysicalDisk`r`n" ` + "mode = single`r`n" ` + "useEnglishOnly = true`r`n" ` + "index = $metrics_index" $m_logical_disk =` "[perfmon://LogicalDisk]`r`n" ` + "counters = Free Megabytes;% Free Space`r`n" ` + "instances = *`r`n" ` + "interval = 30`r`n" ` + "object = LogicalDisk`r`n" ` + "mode = single`r`n" ` + "useEnglishOnly = true`r`n" ` + "index = $metrics_index" $m_network =` "[perfmon://Network]`r`n" ` + "counters = Bytes Received/sec;Bytes Sent/sec;Packets Received/sec;Packets Sent/sec;Packets Received Errors;Packets Outbound Errors`r`n" ` + "instances = *`r`n" ` + "interval = 30`r`n" ` + "mode = single`r`n" ` + "object = Network Interface`r`n" ` + "useEnglishOnly = true`r`n" ` + "index = $metrics_index" $m_system =` "[perfmon://System]`r`n" ` + "counters = Processor Queue Length;Threads`r`n" ` + "instances = *`r`n" ` + "interval = 30`r`n" ` + "object = System`r`n" ` + "mode = single`r`n" ` + "useEnglishOnly = true`r`n" ` + "index = $metrics_index" $m_process =` "[perfmon://Process]`r`n" ` + "counters = % Processor Time;% User Time;% Privileged Time`r`n" ` + "instances = *`r`n" ` + "interval = 30`r`n" ` + "object = Process`r`n" ` + "mode = single`r`n" ` + "useEnglishOnly = true`r`n" ` + "index = $metrics_index" # Inputs.conf options for the supported Windows EventLogs $eventlog_options =` "checkpointInterval = 10`r`n" ` + "current_only = 1`r`n" ` + "disabled = 0`r`n" ` + "start_from = oldest" # extract os & ip info # > WIN 7 #$os_info = Get-CimInstance Win32_OperatingSystem | Select-Object Caption, Version # =WIN 7 $os_info = Get-WmiObject Win32_OperatingSystem | Select-Object Caption, Version $ip_info = Test-Connection -ComputerName $env:computername -count 1 | Select-Object IPV4Address # Add OS & ip info as dimension # If these dimensions are to be updated, user has to rerun this script $dims = $dimensions + " os::" + "`"" + $os_info.Caption + "`"" $dims = $dims + " os_version::" + $os_info.Version $dims = $dims + " ip::" + "`"" + $ip_info.IPV4Address.IPAddressToString + "`"" $dims = $dims + ' entity_type::Windows_Host' $metrics = $metrics_param -split ',' $log_sources = $log_sources_param -split ',' echo "# *** Configure Metrics Logs collected ***" > $inputconfTemp For ($i=0; $i -lt $metrics.Length; $i++) { $m_name = "m_" + $metrics[$i] Get-Variable -Name $m_name -ValueOnly -ErrorAction 'Ignore' >> $inputconfTemp # Add dimensions echo "_meta = $dims" >> $inputconfTemp echo "`n" >> $inputconfTemp } For ($i=0; $i -lt $log_sources.Length; $i++) { if([string]::IsNullOrEmpty($log_sources[$i])) { continue } # split log source into source and sourcetype $logsource = $log_sources[$i] -split '%' if ($logsource.Length -ne 2) { continue } $log_source = $($logsource[0]) $log_sourcetype = $($logsource[1]) if ($log_sourcetype -eq 'WinEventLog') { #Get-Variable -Name $log_source -ValueOnly -ErrorAction 'Ignore' >> $inputconfTemp echo "[WinEventLog://$log_source]" >> $inputconfTemp echo "$eventlog_options" >> $inputconfTemp echo "`r`n" >> $inputconfTemp } else { echo "[monitor://$log_source]" >> $inputconfTemp echo "sourcetype = $log_sourcetype" >> $inputconfTemp echo "disabled = false" >> $inputconfTemp if ($log_sourcetype -eq 'collectd' -Or $log_sourcetype -eq 'uf') { echo "index = _internal" >> $inputconfTemp } echo "`r`n" >> $inputconfTemp } } if (Test-Path -Path $splunkpath"inputs.old" ) {Remove-Item -Path $splunkpath"inputs.old"} if (Test-Path -Path $splunkpath"inputs.conf" ) {Rename-Item -Path $splunkpath"inputs.conf" -NewName $splunkpath"inputs.old"} if (Test-Path -Path $splunkpath"inputs.conf" ) {Remove-Item -Path $splunkpath"inputs.conf"} Rename-Item -Path $inputconfTemp -NewName $splunkpath"inputs.conf" TO Roll-out with Forwarder Manager: Create the new Applications on the Forwarder Manager Name: ManagedInfrastructure etc/deployment-apps/ManagedInfrastructure/bin/install.path etc/deployment-apps/ManagedInfrastructure/bin/install.ps1 etc/deployment-apps/ManagedInfrastructure/local/inputs.conf Name: Restart_Only etc/deployment-apps/Restart_Only Set Ownership of new files chown -R splunk:splunk /opt/splunk/etc/deployment-apps/ManagedInfrastructure/ chown -R splunk:splunk /opt/splunk/etc/deployment-apps/Restart_Only install.path: C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -ExecutionPolicy RemoteSigned -Command "& 'C:\Program Files\SplunkUniversalForwarder\etc\apps\ManagedInfrastructure\bin\install.ps1' inputs.conf [script://.\bin\install.path] interval = -1 Push out with the Manager, then Push out Restart_Only app. ... View more

100% working, thank you! ... View more

Awesome, this appears to be the missing link. ... View more