What is the exact text of the error message(s) you receive?
I see below error in search head captain and status show as red.
Root Cause(s):
The percentage of high priority searches delayed (11%) over the last 24 hours is very high and exceeded the red thresholds (10%) on this Splunk instance. Total Searches that were part of this percentage=100. Total delayed Searches=11
Last 50 related messages:
Did anything else change besides the version of Splunk?
We have not changed anything after version upgrade.
What is you Splunk environment like (single instance, cluster, etc.)?
Our environment is in cluster..4 indexer 3 search head 1 deployer 1 matster
When you say the custom alerts are "not working" what exactly do you mean? What are the symptoms? Are they running at all? Have you checked the search log for the alerts?
Custom alert conditions are not working. I have investigated scheduler log. I could see that status is success alert action is blank.one user raised the concern that alert is not triggering when the custom condition are met. Later we realized that its not working with all the alerts which are using custom condition.
10-31-2020 08:10:07.566 +0000 INFO SavedSplunker - savedsearch_id="XXX;search; alert", search_type="", user="XXX", app="search", savedsearch_name="XXXX alert", priority=default, status=success, digest_mode=1, scheduled_time=1604131800, window_time=0, dispatch_time=1604131805, run_time=1.785, result_count=1015, alert_actions="", sid="scheduler__smadan__search__RMD5ab6a869ca92dbacc_at_1604131800_63960_638683B3-25D9-4D2A-AF2E-4E43362FDBFA", suppressed=0, thread_id="AlertNotifierWorker-0", workload_pool=""
