All Apps and Add-ons

Why is forwarding Windows performance logs not working?

mochocki
Explorer

I have two windows serwers (srv_iis/srv_sql) infrastructure with indexer deployed on srv_iis and forwarder deployed on srv_sql. What I want to achieve is to forward performance counters from srv_sql server to srv_iis.
Facts:
- indexing on the indexer server (srv_iis) works fine
- forwarding event log srv_sql -> srv_iis works fine
- forwarding performance counters the same way is NOT WORKING AT ALL
- WMI is not an option since this is not AD setup (just a workgroup)
- network communication between servers is ok (telnet on mngmt port works fine)
- My Splunk version is 6.5.2
- I tried to deploy this docs.splunk.com/Documentation/MSApp/latest/MSInfra/AbouttheSplunkAppforMSInfrastructure and it is not working as well (perfmon index on the indexer is empty)

inputs.conf from forwarder:

[default]
host = srv_sql

[WinEventLog://Application] <---- this works fine
disabled = 0
index = perf

[perfmon://LocalMainMemory]
interval = 5
object = Memory
counters = Committed Bytes; Available Bytes; % Committed Bytes In Use
disabled = 0
index = perf

[perfmon://Available Memory]
counters = *
interval = 10
object = Memory
index = perf

outputs.fonf from forwarder:

[tcpout]
defaultGroup = default-autolb-group

[tcpout:default-autolb-group]
server = srv_iis:9997

[tcpout-server://srv_iis:9997]
0 Karma
1 Solution

adonio
Ultra Champion

Please relay on Windows TA pre built inputs for example:
[perfmon://Memory]
counters = Page Faults/sec; Available Bytes; Committed Bytes; Commit Limit; Write Copies/sec; Transition Faults/sec; Cache Faults/sec; Demand Zero Faults/sec; Pages/sec; Pages Input/sec; Page Reads/sec; Pages Output/sec; Pool Paged Bytes; Pool Nonpaged Bytes; Page Writes/sec; Pool Paged Allocs; Pool Nonpaged Allocs; Free System Page Table Entries; Cache Bytes; Cache Bytes Peak; Pool Paged Resident Bytes; System Code Total Bytes; System Code Resident Bytes; System Driver Total Bytes; System Driver Resident Bytes; System Cache Resident Bytes; % Committed Bytes In Use; Available KBytes; Available MBytes; Transition Pages RePurposed/sec; Free & Zero Page List Bytes; Modified Page List Bytes; Standby Cache Reserve Bytes; Standby Cache Normal Priority Bytes; Standby Cache Core Bytes; Long-Term Average Standby Cache Lifetime (s)
disabled = 0
interval = 10
object = Memory
useEnglishOnly=true
index = perfmon (or index=perf)
download the TA here: https://splunkbase.splunk.com/app/742/
navigate to ...\apps\splunk_TA_Windows\default and check all inputs at inupts.conf
create a local directory and copy the needed inputs.
modify disabled = 1 to disabled = 0 for the inputs you wish to enable
Cheers

View solution in original post

adonio
Ultra Champion

Please relay on Windows TA pre built inputs for example:
[perfmon://Memory]
counters = Page Faults/sec; Available Bytes; Committed Bytes; Commit Limit; Write Copies/sec; Transition Faults/sec; Cache Faults/sec; Demand Zero Faults/sec; Pages/sec; Pages Input/sec; Page Reads/sec; Pages Output/sec; Pool Paged Bytes; Pool Nonpaged Bytes; Page Writes/sec; Pool Paged Allocs; Pool Nonpaged Allocs; Free System Page Table Entries; Cache Bytes; Cache Bytes Peak; Pool Paged Resident Bytes; System Code Total Bytes; System Code Resident Bytes; System Driver Total Bytes; System Driver Resident Bytes; System Cache Resident Bytes; % Committed Bytes In Use; Available KBytes; Available MBytes; Transition Pages RePurposed/sec; Free & Zero Page List Bytes; Modified Page List Bytes; Standby Cache Reserve Bytes; Standby Cache Normal Priority Bytes; Standby Cache Core Bytes; Long-Term Average Standby Cache Lifetime (s)
disabled = 0
interval = 10
object = Memory
useEnglishOnly=true
index = perfmon (or index=perf)
download the TA here: https://splunkbase.splunk.com/app/742/
navigate to ...\apps\splunk_TA_Windows\default and check all inputs at inupts.conf
create a local directory and copy the needed inputs.
modify disabled = 1 to disabled = 0 for the inputs you wish to enable
Cheers

adonio
Ultra Champion

Hi mochocki,
can you verify its index=perfmon and not index=perf as shows in your code?
or you created that index for the data?
also, are you usinf the windows TA? https://splunkbase.splunk.com/app/742/

0 Karma

mochocki
Explorer

Hi,
Index perfmon comes from MSApp. Index perf comes from my configuration. Both do not contain any performance entries.

0 Karma

adonio
Ultra Champion

are you using the rebuilt perfmon inputs from the TA? can yuo try and place this in your inputs.conf and check?
[perfmon://Memory]
counters = Page Faults/sec; Available Bytes; Committed Bytes; Commit Limit; Write Copies/sec; Transition Faults/sec; Cache Faults/sec; Demand Zero Faults/sec; Pages/sec; Pages Input/sec; Page Reads/sec; Pages Output/sec; Pool Paged Bytes; Pool Nonpaged Bytes; Page Writes/sec; Pool Paged Allocs; Pool Nonpaged Allocs; Free System Page Table Entries; Cache Bytes; Cache Bytes Peak; Pool Paged Resident Bytes; System Code Total Bytes; System Code Resident Bytes; System Driver Total Bytes; System Driver Resident Bytes; System Cache Resident Bytes; % Committed Bytes In Use; Available KBytes; Available MBytes; Transition Pages RePurposed/sec; Free & Zero Page List Bytes; Modified Page List Bytes; Standby Cache Reserve Bytes; Standby Cache Normal Priority Bytes; Standby Cache Core Bytes; Long-Term Average Standby Cache Lifetime (s)
disabled = 0
interval = 10
object = Memory
useEnglishOnly=true
index = perfmon (or index=perf)

0 Karma

mochocki
Explorer

It works! Thank you!
Still do not understand what was wrong. The only difference I see is useEnglishOnly=true.
My Windows locale is Polish - is that the problem?

0 Karma

adonio
Ultra Champion

there is also a difference in the stanza [perfmon://LocalMainMemory] - yours
compare to [perfmon://Memory] - prebuilt Windows TA
will place in the answer section
cheers

0 Karma
Get Updates on the Splunk Community!

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...

March Community Office Hours Security Series Uncovered!

Hello Splunk Community! In March, Splunk Community Office Hours spotlighted our fabulous Splunk Threat ...

Stay Connected: Your Guide to April Tech Talks, Office Hours, and Webinars!

Take a look below to explore our upcoming Community Office Hours, Tech Talks, and Webinars in April. This post ...