Hi
We recently upgraded our Splunk instance from 4.0.10 to 4.1.4. After the upgrade we are seeing the following error being generated from one of the charts using a summary-indexed query:
Error in 'IndexScopedSearch': The search failed. More than 125000 events found at time 1286150400.
The query is of the form:
index=summary s_name="blah" | stats count as s_count by orig_host, field1, _time |
timechart sum(s_count) by orig_host
This chart worked fine in 4.0.10. Thanks for your help.
Ranga
Looks like you are producing too many events on a single timestamp within your summary index saved search.
1286150400 is 2010-10-04 00:00.00
Are you producing your summary index events with sistats
? If so, you should look very carefully at which stats functions you are using. You can actually end up with more events in your summary index than you are summarizing depending on your field/function combinations.
Update:
If you haven't found this already, the following is helpful information in understanding the issue at hand. (I'm fairly certain this is the same issue, but I could be wrong.)
@dmlee, you may want to open a new question. This question is specifically referring to a change in behavior after an upgrade; but there is a more generic question here: How to avoid 125000 events limit in my summary index? If you ask a question like that you me able to get some additional attention. I don't have a good answer, especially without seeing the related searches. I added a link of interest to my answer that may be helpful too.
@sranga, sorry for getting back so late, I must have missed your comment. While you can use addinfo
and collect
and all that, you don't have to, and that's not what I'm suggesting that you change. I'm simply pointing out that you can change your search to use | stats
instead of using | sistats
the si
-prefixed search operators sometimes give you extra value add and generally lets you do less thinking upfront, but I've found that blindly choosing sistats
over stats
can lead to other problems if you aren't paying attention to how much output it creates.
we also have this problem, is there any way to extend the limition ? because we can't find other way to reduce events on a single timestamp.
Thanks. If I don't use sistats, how do I summary index my data? Are you suggesting that I manually configure the summary index using addinfo
and collect
?
For a simple count like that, I'm not sure that sistats
buys you anything that plain stats
wouldn't do. You do have to watch out for how you name your fields, but that's not a big deal. It would also be good to confirm that your scheduled interval and search time range aren't overlapping or anything like that. That could certainly lead to too many summary events as well.
Thanks. Yes, I use sistats. The query specified in the question is the one I use in my summary index and it is of the form: index=summary s_name="blah" | sistats count as s_count by orig_host, field1, _time