Differences in eventcount results and 'real' search counts.

[pdjhh]

Hi.

A site we are on has attemtped to migrate data from one splunk cluster to another. We've come in late to help and have fixed most things up but they are noticing a difference in their eventcount type searches and real event searches. So index=* | stats count by index gives one set of numbers but eventcount summarize=false index=* gives quite different numbers, an amount less.

I'm thinking the metadata values have been messed up during their copy activities and wonder if they can be rebuilt?

Thanks.

[pdjhh · Accepted solution]

We've had Splunk look into this with us and after a bit of time they are coming back basically saying not to use eventcount in clustered environments. I'll quote so that you can make your own interpretation of the words for now:

Splunk: "Eventcount will check with all the buckets including the excessive/replicated and primary buckets and is not recommended to be used for event count in the index cluster env"

Me: "are you saying the recommendation is not to use eventcount in a clustered environment? "

Splunk: "Yes. eventcount is not recommended to count events for comparison in a clustered environment. Feedback has been sent to document team to update the Splunk docs. "

[pdjhh]

Focussing on one index then it's the same:

index=xyz | stats count as "Events" gives a result of 63 million and something

| eventcount index=xyz gives a result of 64 million and something

How can these differ? Especially with the eventcount figure being more?

Interestingly | metadata type=hosts index=xyz | stats sum(totalCount) gives the exact same value as what I'm calling the 'real' search of index=xyz | stats count. A similar tstats gives the same figure as well so it's only the eventcount search that's returning a different value..