We recently upgraded our Splunk instance from 4.0.10 to 4.1.4. After the upgrade we are seeing the following error being generated from one of the charts using a summary-indexed query:
Error in 'IndexScopedSearch': The search failed. More than 125000 events found at time 1286150400.
The query is of the form:
index=summary s_name="blah" | stats count as s_count by orig_host, field1, _time | timechart sum(s_count) by orig_host
This chart worked fine in 4.0.10. Thanks for your help.
Looks like you are producing too many events on a single timestamp within your summary index saved search.
1286150400 is 2010-10-04 00:00.00
Are you producing your summary index events with
sistats? If so, you should look very carefully at which stats functions you are using. You can actually end up with more events in your summary index than you are summarizing depending on your field/function combinations.
If you haven't found this already, the following is helpful information in understanding the issue at hand. (I'm fairly certain this is the same issue, but I could be wrong.)
@dmlee, you may want to open a new question. This question is specifically referring to a change in behavior after an upgrade; but there is a more generic question here: How to avoid 125000 events limit in my summary index? If you ask a question like that you me able to get some additional attention. I don't have a good answer, especially without seeing the related searches. I added a link of interest to my answer that may be helpful too.
@sranga, sorry for getting back so late, I must have missed your comment. While you can use
collect and all that, you don't have to, and that's not what I'm suggesting that you change. I'm simply pointing out that you can change your search to use
| stats instead of using
| sistats the
si-prefixed search operators sometimes give you extra value add and generally lets you do less thinking upfront, but I've found that blindly choosing
stats can lead to other problems if you aren't paying attention to how much output it creates.
For a simple count like that, I'm not sure that
sistats buys you anything that plain
stats wouldn't do. You do have to watch out for how you name your fields, but that's not a big deal. It would also be good to confirm that your scheduled interval and search time range aren't overlapping or anything like that. That could certainly lead to too many summary events as well.
Thanks. Yes, I use sistats. The query specified in the question is the one I use in my summary index and it is of the form: index=summary s_name="blah" | sistats count as s_count by orig_host, field1, _time