Why does metrics.log search on a clustered environment display double the license usage value from the license master?

[DavidHourani]

Hello Splunkers,

I am trying to understand the results of the following command on a Search Head (SH) in a clustered environment (Search Head cluster + indexer cluster):

index="_internal" source="*metrics.log" per_index_thruput  | eval GB=kb/(1024*1024) | timechart span=1d sum(GB) | convert ctime(_time) as timestamp

Using it on a standalone instance this gives me the same values as my license usage. When I use it on a SH in a clustered environment, it's giving me double the license usage value from the license master.

Any idea why this could be the case? Does replication have anything to do with it?

Regards,
David

[twinspop]

Metrics logs are produced by all Splunk instances. If you have forwarders sending logs to an indexer, for example, you will potentially have 2 sets of metrics logs for any one category/series. You can get around this problem in 1 of 2 ways:

1) Isolate your search to your indexer(s) using host=your_indexer. The drawback here is that metrics are samples. By default, splunk only tracks the top 10 in each category. You can adjust this with maxseries under the metrics stanza in limits.conf.

2) Isolate your search to only the forwarders. One way to do this would be to EXCLUDE the indexers from your search: host!=your_indexer. Splunk still only tracks the top 10, but each forwarder has their own top 10, and you're more likely to get a complete(r) picture.

[twinspop]

Also, see here for more Metrics goodness:

http://docs.splunk.com/Documentation/Splunk/6.5.0/Troubleshooting/Aboutmetricslog