Why are we getting different events for the same sourcetype when searching for the same time period?

[abhijit_mhatre]

There are sourcetypes which are returning different number of events for the same time period.

Example:
sourcetype="ABC" returns 90,000 events for the last 15 mins
sourcetype="ABC" returns 94,000 events for the last 15 mins

ABC is the same sourcetype used while searching events.

[esix_splunk · Accepted solution]

Whats your full search on those? Are you including an index name, e.g.

index=indexforsourcetypeabc sourcetype=abc

VS a wildcard search..

sourcetype=abc

Depending on the user permissions, these could be returning different results from different indexes.

Also, is this in a distributed environment? Is the same search from a single user returning different results?

[abhijit_mhatre]

It is a wildcard search in a distributed environment.
Yes, it is a same search from a single user returning different results.

[esix_splunk]

You can should try looking at the search log between the different searches and see if you have an indexer that's not returning results or perhaps you have a underlying performance issue.

You can try running a search like

index=abcindex sourcetype="ABC" | stats count by splunk_server

Take a look at that and see what your event count is across your indexers. They should be balanced fairly evenly. Where events aren't balanced should point to either new servers or potential issues.
Without knowing more about your environment, it's hard to help.

[abhijit_mhatre]

Thanks for your response.
I tried running the query using index & count by splunk_server, but still events are not balanced. There might be a performance issue of the sourcetypes.