Splunk Search

Splunk search returning inconsistent results

johnboldt
Explorer

The following search which spans an hour returns 10,000 events which are all included in the final time bucket (ie 10:59).

sourcetype="sourcetype" earliest=1/5/2011:10:00:00 latest=1/5/2011:11:00:00 | fields Application, Level, ThreadId, Message, host, cluster | search Application="*" Level="Activity"

However, this search which spans one minute returns 15,000 events:

sourcetype="sourcetype" earliest=1/5/2011:10:00:00 latest=1/5/2011:10:01:00 | fields Application, Level, ThreadId, Message, host, cluster | search Application="*" Level="Activity"

We haven't made any changes to limits.conf and have just started to see this problem. One recent change was specifying LINE_BREAKER in props.conf. Any ideas on the inconsistency?

0 Karma

sideview
SplunkTrust
SplunkTrust

That seems strange.
But why are you pulling out the Application="*" and Level="Activity" into a second search clause? The way you have it the first search clause will pull a certain amount of events off disk (possibly quite a large number of events), and then the second search clause will filter out some of those events (again possibly a huge number of events). It's possible the search is so inefficient that it's hitting some rather high limit somewhere. Is it a very slow search as well? As the search is progressing does the UI say that the 'scanned events' are markedly higher than 'events'?

At any rate I'd advise putting those terms in the main search clause. Rest assured that the fields exist will be extracted and will exist even before the fields clause refers to them.

0 Karma

johnboldt
Explorer

Nick - thanks for the suggestion. This was one of the first searches we incorporated and the second search clause was definitely a newbie mistake. I moved the Level="Activity" to the first search clause and it works fine now.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...