Splunk Search

Why are we getting different events for the same sourcetype when searching for the same time period?

abhijit_mhatre
Path Finder

There are sourcetypes which are returning different number of events for the same time period.

Example:
sourcetype="ABC" returns 90,000 events for the last 15 mins
sourcetype="ABC" returns 94,000 events for the last 15 mins

ABC is the same sourcetype used while searching events.

0 Karma
1 Solution

esix_splunk
Splunk Employee
Splunk Employee

Whats your full search on those? Are you including an index name, e.g.

index=indexforsourcetypeabc sourcetype=abc

VS a wildcard search..

sourcetype=abc

Depending on the user permissions, these could be returning different results from different indexes.

Also, is this in a distributed environment? Is the same search from a single user returning different results?

View solution in original post

0 Karma

esix_splunk
Splunk Employee
Splunk Employee

Whats your full search on those? Are you including an index name, e.g.

index=indexforsourcetypeabc sourcetype=abc

VS a wildcard search..

sourcetype=abc

Depending on the user permissions, these could be returning different results from different indexes.

Also, is this in a distributed environment? Is the same search from a single user returning different results?

0 Karma

abhijit_mhatre
Path Finder

It is a wildcard search in a distributed environment.
Yes, it is a same search from a single user returning different results.

0 Karma

esix_splunk
Splunk Employee
Splunk Employee

You can should try looking at the search log between the different searches and see if you have an indexer that's not returning results or perhaps you have a underlying performance issue.

You can try running a search like

index=abcindex sourcetype="ABC" | stats count by splunk_server

Take a look at that and see what your event count is across your indexers. They should be balanced fairly evenly. Where events aren't balanced should point to either new servers or potential issues.
Without knowing more about your environment, it's hard to help.

0 Karma

abhijit_mhatre
Path Finder

Thanks for your response.
I tried running the query using index & count by splunk_server, but still events are not balanced. There might be a performance issue of the sourcetypes.

0 Karma
*NEW* Splunk Love Promo!
Snag a $25 Visa Gift Card for Giving Your Review!

It's another Splunk Love Special! For a limited time, you can review one of our select Splunk products through Gartner Peer Insights and receive a $25 Visa gift card!

Review:





Or Learn More in Our Blog >>