All Apps and Add-ons

Same counter prouduce different values (magnitude)

Path Finder

I collect performance counters from multiple servers, using this configuration:

[PERFMON:Matchers]
counters = % User Time
disabled = 0
index = default
instances = HashMatcher;HashMatcher#1;HashMatcher#2;HashMatcher#3;HashMatcher#4;HashMatcher#5;HashMatcher#6;HashMatcher#7;HashMatcher#8;HashMatcher#9
interval = 30
object = Process

I have 6 servers from which I collect the data.

Search is:

"collection=Matchers" | chart max(Value) by host

or

"collection=Matchers" | timechart span=1m sum(Value) by host

And the results are, respectively:

alt text

and

alt text

So my questions would be: WHY does magnitude of this values differ so much? I can guess all night long, but what exactly is going on?

BTW, I tried different counters, and problem isn't related to the host - different counter produces problem on only access4, for example.

Help!

0 Karma
1 Solution

Path Finder

I have to try to answer my own question, since it seems that I found the problem. It is really interesting.

I was changing the pefmon.conf manually with the editor. I was also checking if everything is OK form web GUI of the splunk. It was. After 3 servers, I decided that restarting the server isn't necessary, since GUI was displaying new data after perfmon.conf is saved and page refreshed under web GUI. But, it wasn't the case.

BTW, I also figured out that restarting splunkd through services doesn't work either. So, one needs to go through web GUI to restart it.

View solution in original post

0 Karma

Path Finder

I have to try to answer my own question, since it seems that I found the problem. It is really interesting.

I was changing the pefmon.conf manually with the editor. I was also checking if everything is OK form web GUI of the splunk. It was. After 3 servers, I decided that restarting the server isn't necessary, since GUI was displaying new data after perfmon.conf is saved and page refreshed under web GUI. But, it wasn't the case.

BTW, I also figured out that restarting splunkd through services doesn't work either. So, one needs to go through web GUI to restart it.

View solution in original post

0 Karma

Path Finder

Restarting server did some trick. When I looked into raw event data, there were something from that server that wasn't even configured (any more).

0 Karma

Path Finder

Very good question - events look OK! I mean, their data is OK.

0 Karma

SplunkTrust
SplunkTrust

What do the events look like?

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!