Same counter prouduce different values (magnitude)

[mosmondor]

I collect performance counters from multiple servers, using this configuration:

[PERFMON:Matchers]
counters = % User Time
disabled = 0
index = default
instances = HashMatcher;HashMatcher#1;HashMatcher#2;HashMatcher#3;HashMatcher#4;HashMatcher#5;HashMatcher#6;HashMatcher#7;HashMatcher#8;HashMatcher#9
interval = 30
object = Process

I have 6 servers from which I collect the data.

Search is:

"collection=Matchers" | chart max(Value) by host

or

"collection=Matchers" | timechart span=1m sum(Value) by host

And the results are, respectively:

and

So my questions would be: WHY does magnitude of this values differ so much? I can guess all night long, but what exactly is going on?

BTW, I tried different counters, and problem isn't related to the host - different counter produces problem on only access4, for example.

Help!

[mosmondor · Accepted solution]

I have to try to answer my own question, since it seems that I found the problem. It is really interesting.

I was changing the pefmon.conf manually with the editor. I was also checking if everything is OK form web GUI of the splunk. It was. After 3 servers, I decided that restarting the server isn't necessary, since GUI was displaying new data after perfmon.conf is saved and page refreshed under web GUI. But, it wasn't the case.

BTW, I also figured out that restarting splunkd through services doesn't work either. So, one needs to go through web GUI to restart it.

[mosmondor]

Restarting server did some trick. When I looked into raw event data, there were something from that server that wasn't even configured (any more).

[mosmondor]

Very good question - events look OK! I mean, their data is OK.

[martin_mueller]

What do the events look like?