topic Re: IndexScopedSearch Error in Splunk Search

IndexScopedSearch Error

sranga — Thu, 07 Oct 2010 05:48:22 GMT

We recently upgraded our Splunk instance from 4.0.10 to 4.1.4. After the upgrade we are seeing the following error being generated from one of the charts using a summary-indexed query:

Error in 'IndexScopedSearch': The search failed. More than 125000 events found at time 1286150400.

The query is of the form:

index=summary s_name="blah" | stats count as s_count by orig_host, field1, _time | 
timechart sum(s_count) by orig_host

This chart worked fine in 4.0.10. Thanks for your help.

Ranga

Re: IndexScopedSearch Error

Lowell — Thu, 07 Oct 2010 06:51:24 GMT

Looks like you are producing too many events on a single timestamp within your summary index saved search.

1286150400 is 2010-10-04 00:00.00

Are you producing your summary index events with sistats? If so, you should look very carefully at which stats functions you are using. You can actually end up with more events in your summary index than you are summarizing depending on your field/function combinations.

Update:

If you haven't found this already, the following is helpful information in understanding the issue at hand. (I'm fairly certain this is the same issue, but I could be wrong.)

http://answers.splunk.com/questions/303/whats-max-events-i-can-have-timestamped-with-a-particular-second-millisecond

Re: IndexScopedSearch Error

sranga — Mon, 28 Sep 2020 09:18:53 GMT

Thanks. Yes, I use sistats. The query specified in the question is the one I use in my summary index and it is of the form: index=summary s_name="blah" | sistats count as s_count by orig_host, field1, _time

Re: IndexScopedSearch Error

Lowell — Fri, 08 Oct 2010 20:35:20 GMT

For a simple count like that, I'm not sure that sistats buys you anything that plain stats wouldn't do. You do have to watch out for how you name your fields, but that's not a big deal. It would also be good to confirm that your scheduled interval and search time range aren't overlapping or anything like that. That could certainly lead to too many summary events as well.

Re: IndexScopedSearch Error

sranga — Sat, 09 Oct 2010 00:54:21 GMT

Thanks. If I don't use sistats, how do I summary index my data? Are you suggesting that I manually configure the summary index using addinfo and collect?

Re: IndexScopedSearch Error

dmlee — Thu, 06 Jan 2011 19:28:26 GMT

we also have this problem, is there any way to extend the limition ? because we can't find other way to reduce events on a single timestamp.

Re: IndexScopedSearch Error

Lowell — Thu, 06 Jan 2011 22:45:49 GMT

@sranga, sorry for getting back so late, I must have missed your comment. While you can use addinfo and collect and all that, you don't have to, and that's not what I'm suggesting that you change. I'm simply pointing out that you can change your search to use | stats instead of using | sistats the si-prefixed search operators sometimes give you extra value add and generally lets you do less thinking upfront, but I've found that blindly choosing sistats over stats can lead to other problems if you aren't paying attention to how much output it creates.

Re: IndexScopedSearch Error

Lowell — Thu, 06 Jan 2011 22:56:56 GMT

@dmlee, you may want to open a new question. This question is specifically referring to a change in behavior after an upgrade; but there is a more generic question here: How to avoid 125000 events limit in my summary index? If you ask a question like that you me able to get some additional attention. I don't have a good answer, especially without seeing the related searches. I added a link of interest to my answer that may be helpful too.