Deployment Architecture

How to distribute Distributed Search configuration using a deployer for a Search Head Cluster?

att35
Builder

Hi,

We recently set up a SH Cluster which includes 3 members and one deployer. Basic replication seems to be working fine(tested by creating a dashboard on one member), but running into issues when deploying configuration changes. What are the best practices when it comes to deploy a system configuration, e.g. distributed search peer's, from the Deployer to all the SH members?

If I understood the steps correctly, the only way to deploy anything from a deployer is to create an app under /opt/splunk/etc/shcluster/apps.
For this, I created a new folder called "configuration" and copied distsearch.conf from /opt/splunk/etc/system/local/distsearch.conf

Deployment was initiated using splunk apply shcluster-bundle. I can see the changes were accepted on the SH Member under /opt/splunk/etc/apps/configuration, but SH member is still unable to search any peer. Most likely these changes did not take effect.
Is this a wrong way to deploy any system changes using deployer?

Please advise.

Thanks,

~Abhi

0 Karma
1 Solution

jkat54
SplunkTrust
SplunkTrust

You actually configure each search head independently when it comes to distsearch.conf. You dont do it by app.

/opt/splunk/etc/system/local/distsearch.conf <--- will always take precedence over any distsearch.conf because it gets the highest priority

http://docs.splunk.com/Documentation/Splunk/6.2.0/Admin/Wheretofindtheconfigurationfiles

This means you MUST configure it on each search head independently and not via deployment app from the deployer:

http://docs.splunk.com/Documentation/Splunk/6.0/admin/Distsearchconf

To set custom configurations, place a distsearch.conf in $SPLUNK_HOME/etc/system/local/.

For examples, see distsearch.conf.example. You must restart Splunk to enable configurations.

View solution in original post

0 Karma

jkat54
SplunkTrust
SplunkTrust

You actually configure each search head independently when it comes to distsearch.conf. You dont do it by app.

/opt/splunk/etc/system/local/distsearch.conf <--- will always take precedence over any distsearch.conf because it gets the highest priority

http://docs.splunk.com/Documentation/Splunk/6.2.0/Admin/Wheretofindtheconfigurationfiles

This means you MUST configure it on each search head independently and not via deployment app from the deployer:

http://docs.splunk.com/Documentation/Splunk/6.0/admin/Distsearchconf

To set custom configurations, place a distsearch.conf in $SPLUNK_HOME/etc/system/local/.

For examples, see distsearch.conf.example. You must restart Splunk to enable configurations.

0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

You need to create Folder structure like this on Deployer /opt/splunk/etc/apps/configuration/local/ and then copy distsearch.conf in that folder. After that push the bundle from deployer.

Thanks,
Harshil

somesoni2
Revered Legend
  1. Check if your search heads restarted after receiving new distsearch.conf.
  2. If they did, check the status of search peers from Settings->Distributed Search->Search peers. The peers might be saying failed status due to authentication. Open each one of them and provide admin credentials.
0 Karma

att35
Builder

Thank you.

We checked the members and they did in fact need the credentials to be re-entered. Once that was done it was all set and each member was able to query the pool.

0 Karma
Get Updates on the Splunk Community!

New Splunk Observability innovations: Deeper visibility and smarter alerting to ...

You asked, we delivered. Splunk Observability Cloud has several new innovations giving you deeper visibility ...

Synthetic Monitoring: Not your Grandma’s Polyester! Tech Talk: DevOps Edition

Register today and join TekStream on Tuesday, February 28 at 11am PT/2pm ET for a demonstration of Splunk ...

Instrumenting Java Websocket Messaging

Instrumenting Java Websocket MessagingThis article is a code-based discussion of passing OpenTelemetry trace ...