Splunk search returning inconsistent results

[johnboldt]

The following search which spans an hour returns 10,000 events which are all included in the final time bucket (ie 10:59).

sourcetype="sourcetype" earliest=1/5/2011:10:00:00 latest=1/5/2011:11:00:00 | fields Application, Level, ThreadId, Message, host, cluster | search Application="*" Level="Activity"

However, this search which spans one minute returns 15,000 events:

sourcetype="sourcetype" earliest=1/5/2011:10:00:00 latest=1/5/2011:10:01:00 | fields Application, Level, ThreadId, Message, host, cluster | search Application="*" Level="Activity"

We haven't made any changes to limits.conf and have just started to see this problem. One recent change was specifying LINE_BREAKER in props.conf. Any ideas on the inconsistency?

[sideview]

That seems strange.
But why are you pulling out the Application="*" and Level="Activity" into a second search clause? The way you have it the first search clause will pull a certain amount of events off disk (possibly quite a large number of events), and then the second search clause will filter out some of those events (again possibly a huge number of events). It's possible the search is so inefficient that it's hitting some rather high limit somewhere. Is it a very slow search as well? As the search is progressing does the UI say that the 'scanned events' are markedly higher than 'events'?

At any rate I'd advise putting those terms in the main search clause. Rest assured that the fields exist will be extracted and will exist even before the fields clause refers to them.

[johnboldt]

Nick - thanks for the suggestion. This was one of the first searches we incorporated and the second search clause was definitely a newbie mistake. I moved the Level="Activity" to the first search clause and it works fine now.