Different behavior running Add-on in Search Head Cluster environment

annmarienorcros — Tue, 06 Jul 2021 19:26:40 GMT

I am working on a Splunk Add-on that was developed on stand alone Splunk that I am now testing in a search head cluster environment. The Add-on uses a modular script that sends Splunk search data or notable event data to a 3rd party app when a saved search is matched.

Our Add-on ships 2 saved searches for users to use to verify that the add-on is configured properly with the 3rd party app.

On stand alone Splunk (with Splunk ES also installed) both of these saved searches are triggered when there is a failed Splunk server login that is logged in splunkd:

index=_internal sourcetype=splunkd ERROR UiAuth (for Splunk without Splunk ES)

index=_internal sourcetype=splunkd ERROR UiAuth | `get_event_id` (for Splunk ES)

In the SHC environment, when there is a failed login, we are only seeing one of the two rules triggered (it is not consistent which one is triggered).
Can anyone explain why only one rule would be triggered in SHC environment?
I’m trying to figure out if the SHC is not setup properly, if there is something the Add-on is not doing properly or if there is some other issue?

We are currently testing on Splunk 8.1.0 and Splunk ES 6.1.1, CIM 4.15.0.
Do you think upgrading Splunk to 8.2 would make any difference?

Thanks for any help!

AnnMarie

Re: Different behavior running Add-on in Search Head Cluster environment

codebuilder — Tue, 06 Jul 2021 20:28:58 GMT

How did you install the add-on? To use it on a SHC you would need to use a deployer to push it out, unlike on a standalone where you can install via the web UI.

Re: Different behavior running Add-on in Search Head Cluster environment

annmarienorcros — Tue, 06 Jul 2021 21:07:27 GMT

I installed the add-on to the search heads using the deployer that is running on a separate instance that is not part of the cluster.

topic Different behavior running Add-on in Search Head Cluster environment in Splunk Dev

Different behavior running Add-on in Search Head Cluster environment

Re: Different behavior running Add-on in Search Head Cluster environment

Re: Different behavior running Add-on in Search Head Cluster environment