Search Head Cluster : Duplicate alert action

[andrei1bc]

Hi,

I have a search head cluster with 3 members, that connects to an index cluster of 10 members. Both clusters are running version 7.0 with no replication or performance issues. The search head cluster has an additional alert action installed on all nodes, that performs an API call to an external system.

Whenever an alert is triggered, the action is executed by 2 of the search head cluster members, generating duplicate API calls.

Please assist and thanks in advance.

Regards,
Bogdan

[somesoni2]

Check scheduler logs (index=_internal sourcetype=scheduler) for your alert search to see if the first invocation was completed successfully or not (I believe it should have an event with status=delegated_remote_completion or something). It it was not, they the captain might have assumed it was not completed and fired it again.

[PowerPacked]

Hi @andrei1bc

as somesoni2, mentioned you can look into index=_internal sourcetype=scheduler savedsearch_name=yoursearchname status=success/skipped/delegated/delgated_remote_error/delgated_remote_completion to analyze why its running twice

& also you can look into
index=_introspection sourcetype=splunk_resource_usage component=PerProcess data.process=splunkd data.process_type=search ------- search_props section will list out on what search head the search is running and many other things.

Thanks