Splunk Search

Splunk search returning inconsistent results

johnboldt
Explorer

The following search which spans an hour returns 10,000 events which are all included in the final time bucket (ie 10:59).

sourcetype="sourcetype" earliest=1/5/2011:10:00:00 latest=1/5/2011:11:00:00 | fields Application, Level, ThreadId, Message, host, cluster | search Application="*" Level="Activity"

However, this search which spans one minute returns 15,000 events:

sourcetype="sourcetype" earliest=1/5/2011:10:00:00 latest=1/5/2011:10:01:00 | fields Application, Level, ThreadId, Message, host, cluster | search Application="*" Level="Activity"

We haven't made any changes to limits.conf and have just started to see this problem. One recent change was specifying LINE_BREAKER in props.conf. Any ideas on the inconsistency?

0 Karma

sideview
SplunkTrust
SplunkTrust

That seems strange.
But why are you pulling out the Application="*" and Level="Activity" into a second search clause? The way you have it the first search clause will pull a certain amount of events off disk (possibly quite a large number of events), and then the second search clause will filter out some of those events (again possibly a huge number of events). It's possible the search is so inefficient that it's hitting some rather high limit somewhere. Is it a very slow search as well? As the search is progressing does the UI say that the 'scanned events' are markedly higher than 'events'?

At any rate I'd advise putting those terms in the main search clause. Rest assured that the fields exist will be extracted and will exist even before the fields clause refers to them.

0 Karma

johnboldt
Explorer

Nick - thanks for the suggestion. This was one of the first searches we incorporated and the second search clause was definitely a newbie mistake. I moved the Level="Activity" to the first search clause and it works fine now.

0 Karma
Get Updates on the Splunk Community!

Splunk Observability Cloud | Unified Identity - Now Available for Existing Splunk ...

Raise your hand if you’ve already forgotten your username or password when logging into an account. (We can’t ...

Index This | How many sides does a circle have?

February 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...