Splunk Search

IndexScopedSearch Error

Path Finder

Hi

We recently upgraded our Splunk instance from 4.0.10 to 4.1.4. After the upgrade we are seeing the following error being generated from one of the charts using a summary-indexed query:

Error in 'IndexScopedSearch': The search failed. More than 125000 events found at time 1286150400.

The query is of the form:

index=summary s_name="blah" | stats count as s_count by orig_host, field1, _time | 
timechart sum(s_count) by orig_host  

This chart worked fine in 4.0.10. Thanks for your help.

Ranga

0 Karma

Super Champion

Looks like you are producing too many events on a single timestamp within your summary index saved search.

1286150400 is 2010-10-04 00:00.00

Are you producing your summary index events with sistats? If so, you should look very carefully at which stats functions you are using. You can actually end up with more events in your summary index than you are summarizing depending on your field/function combinations.

Update:

If you haven't found this already, the following is helpful information in understanding the issue at hand. (I'm fairly certain this is the same issue, but I could be wrong.)

http://answers.splunk.com/questions/303/whats-max-events-i-can-have-timestamped-with-a-particular-se...

0 Karma

Super Champion

@dmlee, you may want to open a new question. This question is specifically referring to a change in behavior after an upgrade; but there is a more generic question here: How to avoid 125000 events limit in my summary index? If you ask a question like that you me able to get some additional attention. I don't have a good answer, especially without seeing the related searches. I added a link of interest to my answer that may be helpful too.

0 Karma

Super Champion

@sranga, sorry for getting back so late, I must have missed your comment. While you can use addinfo and collect and all that, you don't have to, and that's not what I'm suggesting that you change. I'm simply pointing out that you can change your search to use | stats instead of using | sistats the si-prefixed search operators sometimes give you extra value add and generally lets you do less thinking upfront, but I've found that blindly choosing sistats over stats can lead to other problems if you aren't paying attention to how much output it creates.

0 Karma

Communicator

we also have this problem, is there any way to extend the limition ? because we can't find other way to reduce events on a single timestamp.

0 Karma

Path Finder

Thanks. If I don't use sistats, how do I summary index my data? Are you suggesting that I manually configure the summary index using addinfo and collect?

0 Karma

Super Champion

For a simple count like that, I'm not sure that sistats buys you anything that plain stats wouldn't do. You do have to watch out for how you name your fields, but that's not a big deal. It would also be good to confirm that your scheduled interval and search time range aren't overlapping or anything like that. That could certainly lead to too many summary events as well.

0 Karma

Path Finder

Thanks. Yes, I use sistats. The query specified in the question is the one I use in my summary index and it is of the form: index=summary sname="blah" | sistats count as scount by orig_host, field1, _time

0 Karma