https://community.splunk.com/t5/Getting-Data-In/Metrics-data-for-same-time-and-number-of-events-is-different/m-p/315179#M59000 <P>you can use the host fields to understand if a log was generated by IndexerA or IndexerB.<BR /> Bye.<BR /> Giuseppe</P> Sat, 08 Apr 2017 09:22:12 GMT gcusello 2017-04-08T09:22:12Z https://community.splunk.com/t5/Getting-Data-In/Metrics-data-for-same-time-and-number-of-events-is-different/m-p/315176#M58997 <P>I've got a Splunk indexer (call it <STRONG>indexerA</STRONG>) on 6.1.5 which is forwarding logs for specific indexes to another Splunk indexer (call it <STRONG>indexerB</STRONG>) which is on 6.5.2. I ran this search on both using the exact same time period (1 hour from 2:30 to 3:30pm) and got different results:</P> <P>index=_internal source=<EM>metrics</EM> group=per_index_thruput series="winevent_dc_index"<BR /> | rename series as index<BR /> | eval MB=kb/1024<BR /> |stats sum(MB) as MB by index</P> <P>On indexerA the search returned 795.783 megabytes and 3,881 (metrics) events<BR /> On indexerB the search returned 1,192.564 megabytes and 3,996 (metrics) events</P> <P>Net I did a simple <CODE>index=winevent_dc_index | stats count</CODE> on both with the same time-frame to see if the number of indexed log entries matched on the two systems. They did:</P> <P>Events on indexerA winevent_dc_index: 596,399<BR /> Events on indexerB winevent_dc_index: 595,399</P> <P>I then drilled down on one of the records on one box, then the other, and compared the source. They were identical, so nothing is being added.</P> <P>Why are these metrics different?</P> Tue, 29 Sep 2020 13:34:25 GMT https://community.splunk.com/t5/Getting-Data-In/Metrics-data-for-same-time-and-number-of-events-is-different/m-p/315176#M58997 wrangler2x 2020-09-29T13:34:25Z https://community.splunk.com/t5/Getting-Data-In/Metrics-data-for-same-time-and-number-of-events-is-different/m-p/315177#M58998 <P>Hi wrangler2x,<BR /> probably the problem is that in IndexerB there are internal logs of both indexers (splunkd.log metrics.log, ...) related to indexing on winevent_dc_index index, insytead in IndexerA there aren't IndexerB logs.<BR /> Try to exclude IndexerB logs from your search and verify results.<BR /> Bye.<BR /> Giuseppe</P> Tue, 29 Sep 2020 13:34:47 GMT https://community.splunk.com/t5/Getting-Data-In/Metrics-data-for-same-time-and-number-of-events-is-different/m-p/315177#M58998 gcusello 2020-09-29T13:34:47Z https://community.splunk.com/t5/Getting-Data-In/Metrics-data-for-same-time-and-number-of-events-is-different/m-p/315178#M58999 <P>I don't think that should be the case, because this is in the <STRONG>outputs.conf</STRONG> file global <STRONG>tcpout</STRONG> stanza that is on the system sending the logs (IndexerA):</P> <PRE><CODE>forwardedindex.0.whitelist = .* forwardedindex.1.blacklist = _.* </CODE></PRE> <P>that should stop the _internal index and more that starts with an underscore from being sent.</P> <P>I've just been looking at the _internal metrics on IndexerB and there is nothing I can see that tells you what system they originated on, so how would you know this is so and exclude from the results?</P> Fri, 07 Apr 2017 22:46:52 GMT https://community.splunk.com/t5/Getting-Data-In/Metrics-data-for-same-time-and-number-of-events-is-different/m-p/315178#M58999 wrangler2x 2017-04-07T22:46:52Z https://community.splunk.com/t5/Getting-Data-In/Metrics-data-for-same-time-and-number-of-events-is-different/m-p/315179#M59000 <P>you can use the host fields to understand if a log was generated by IndexerA or IndexerB.<BR /> Bye.<BR /> Giuseppe</P> Sat, 08 Apr 2017 09:22:12 GMT https://community.splunk.com/t5/Getting-Data-In/Metrics-data-for-same-time-and-number-of-events-is-different/m-p/315179#M59000 gcusello 2017-04-08T09:22:12Z