All Apps and Add-ons

What config changes are need to ensure DB inputs don't run on non-search-head captain nodes.

Path Finder

Hi All,
We are using Splunk DB Connect 2.3 in a search head clustered environment.
While setting up a new database input, we set up the input on the DBX2 app on the deployer(not part of the search head cluster) and then do copy the splunkappdbconnect app from "/apps/splunk/etc/apps/" to " /apps/splunk/etc/shcluster/apps"
and then run ./splunk apply shcluster-bundle -target https://searchheadclustermember:8089, to push it to search heads.

As per our understanding DB inputs run on the search head captain. But as per logs we can find **"action=modular
inputnotrunningoncaptain"**, i.e. db inputs aren't always running on captain.
Does Splunk 1st queries each of the search head peer to find which one is the captain and then finally runs the input query on captain node? If not, what config changes are need to ensure inputs don't run on non-captain nodes?

Thanks ,
Jincy

0 Karma
1 Solution

SplunkTrust
SplunkTrust

Hello jincy18,
please refer to @martin
mueller comment above.
Use Heavy Forwarder when using DBX, specially in a SHC configuration.

View solution in original post

0 Karma

SplunkTrust
SplunkTrust

Hello jincy18,
please refer to @martin
mueller comment above.
Use Heavy Forwarder when using DBX, specially in a SHC configuration.

View solution in original post

0 Karma

SplunkTrust
SplunkTrust

As per http://docs.splunk.com/Documentation/DBX/2.3.1/DeployDBX/Architectureandperformanceconsiderations#Se... you shouldn't run inputs on a SHC:

Splunk recommends running reports (saved searches), alerts, and lookups on the search head or search head cluster captain, and running inputs and outputs from a forwarder. This is because disruptive search head restarts and reloads are more common, and scheduled or streaming bulk data movements can impact the performance of user searches. Poor user experience, reduced performance, increased configuration replication, unwanted data duplication, or even data loss can result from running inputs and outputs on search head infrastructure. Running inputs and outputs on a search head captain does not provide extra fault tolerance or enhance availability, and is not recommended.

0 Karma

Path Finder

Thank you Martin for your valuable inputs.

We do have a plan to move to HWF along with DBC3 but that will take some time.
Since we are currently using DBC2 in SHC, could you provide some pointers to debug mentioned issue?

Also , in the SHC we do not need to maintain the state of DB Connections and reads as even if a captain goes down, other search heads have the latest state of it and can continue processing. How do we address failover in a HWF configuration ?

0 Karma