Does the srchDiskQuota work for real time searches as well as non-real time? We had a user that appeared to do a * search with a large real time window that almost filled up the disk on a search head.
That also brings up the question: We use index=_internal and sourcetype=searches to tell what searches users are running. How can we tell window time and frames on real time searches and time frames on normal searches?
The classic problem with realtime searches,
if they are triggering alerts, then the alert artifact will stay in the dispatch until the expiration of the alert.
And if the option "one alert per events" is setup instead of "one alert per search" it can de-multiply the issue.
Expectation : realtime alltime is awesome
Reality : Realtime alltime is a loaded gun pointing to your foot.
For the long term solution, you should open a bug/enhancement request in splunk.com to request a way to limit the memory and disk usage of searches.
The traditional interface for this sort of comment is a bug report, btw 🙂 I like bug reports!
Hmm, I don't know offhand and will look into it.