I am looking at setting up alerting in splunk, at the moment I don't know the expected frequency or volumes of alerts, are there any performance issues I should consider. We have a 3 node search cluster, 4* indexers.
Are the searches spread across the searchheads? is it possible to fix them to a single Searchhead?
I use the monitoring console to alert me when scheduled searches are getting skipped, for example. We also use it to alert us when the captain changes (frequent changes might indicate some kind of problem.)
You can also view the length of time of your long running schedules tasks etc.
Thanks, I had considered adding a 4th but was looking to fix this particular set of searches to it. If I am reading the doc correctly I could consider setting a couple to adhoc_searchhead = true then monitor for skipped searches occurring on the remaining 1 (or 2 if I and another) searchheads.