I have been hunting down users in my environment running real-time searches as I thought that they were the root cause, but I have also seen from time to time searches like the ones below lingering around hte Jobs queue. The impact of course is that we start to hit our concurrent search limit
Looking at the logs for these searches, majority of them seem to be stuck in the following loop:
10-11-2016 21:23:18.306 INFO UnifiedSearch - Processed search targeting arguments
10-11-2016 21:23:18.306 INFO DispatchThread - Generating results preview took 1 ms
10-11-2016 21:23:23.306 INFO SearchParser - PARSING: timechart count
10-11-2016 21:23:23.306 INFO bucket - Setting info._summary_maxtimespan = 5s
10-11-2016 21:23:23.306 INFO SearchParser - PARSING: stats count AS query| search query = 10
10-11-2016 21:23:23.306 INFO StatsProcessor - No group-by fields specified, incompatible for high cardinality improvements
The search that the log above came from is the following:
source="SOME_SOURCE_LOG_FILE" | regex _raw=".Error.|.error.|.Exception.|.exception."|timechart count
The problem is the regex
command. Try this search instead:
source="SOME_SOURCE_LOG_FILE" (*error* OR *exception*) | timechart count
Thanks for this. Would you mind elaborating? Why would the regex cause this?
If you run that search for the last 60 minutes, how many events does it return? How many different hosts do they come from?