Different behavior running Add-on in Search Head Cluster environment

[annmarienorcros]

I am working on a Splunk Add-on that was developed on stand alone Splunk that I am now testing in a search head cluster environment. The Add-on uses a modular script that sends Splunk search data or notable event data to a 3rd party app when a saved search is matched.

Our Add-on ships 2 saved searches for users to use to verify that the add-on is configured properly with the 3rd party app.

On stand alone Splunk (with Splunk ES also installed) both of these saved searches are triggered when there is a failed Splunk server login that is logged in splunkd:

index=_internal sourcetype=splunkd ERROR UiAuth                                            (for Splunk without Splunk ES)

index=_internal sourcetype=splunkd ERROR UiAuth | `get_event_id`        (for Splunk ES)

In the SHC environment, when there is a failed login, we are only seeing one of the two rules triggered (it is not consistent which one is triggered).
Can anyone explain why only one rule would be triggered in SHC environment?
I’m trying to figure out if the SHC is not setup properly, if there is something the Add-on is not doing properly or if there is some other issue?

We are currently testing on Splunk 8.1.0 and Splunk ES 6.1.1, CIM 4.15.0.
Do you think upgrading Splunk to 8.2 would make any difference?

Thanks for any help!

AnnMarie

[annmarienorcros]

I installed the add-on to the search heads using the deployer that is running on a separate instance that is not part of the cluster.

[codebuilder]

How did you install the add-on? To use it on a SHC you would need to use a deployer to push it out, unlike on a standalone where you can install via the web UI.