Every once in a while, we will get this message, and I understand it.
I would like to run a periodic search, add it to a summary index, and chart against it so that I can see when it happens historically.
I have read through some of the answers here, but I am wondering if there is just a Splunk search that will spit out the number of concurrent searches happening at that time. Does that exist? Since the error shows a number, I am assuming it is possible?
Thanks!
Try something like this
index=_internal sourcetype=splunkd source=*metrics.log group=search_concurrency "system total"
| timechart max(active_hist_searches) as "Historical Searches" min(active_realtime_searches) as "Real-time Searches" by host