How to distribute Distributed Search configuration using a deployer for a Search Head Cluster?

[att35]

Hi,

We recently set up a SH Cluster which includes 3 members and one deployer. Basic replication seems to be working fine(tested by creating a dashboard on one member), but running into issues when deploying configuration changes. What are the best practices when it comes to deploy a system configuration, e.g. distributed search peer's, from the Deployer to all the SH members?

If I understood the steps correctly, the only way to deploy anything from a deployer is to create an app under /opt/splunk/etc/shcluster/apps.
For this, I created a new folder called "configuration" and copied distsearch.conf from /opt/splunk/etc/system/local/distsearch.conf

Deployment was initiated using splunk apply shcluster-bundle. I can see the changes were accepted on the SH Member under /opt/splunk/etc/apps/configuration, but SH member is still unable to search any peer. Most likely these changes did not take effect.
Is this a wrong way to deploy any system changes using deployer?

Please advise.

Thanks,

~Abhi

[jkat54 · Accepted solution]

You actually configure each search head independently when it comes to distsearch.conf. You dont do it by app.

/opt/splunk/etc/system/local/distsearch.conf <--- will always take precedence over any distsearch.conf because it gets the highest priority

http://docs.splunk.com/Documentation/Splunk/6.2.0/Admin/Wheretofindtheconfigurationfiles

This means you MUST configure it on each search head independently and not via deployment app from the deployer:

http://docs.splunk.com/Documentation/Splunk/6.0/admin/Distsearchconf

To set custom configurations, place a distsearch.conf in $SPLUNK_HOME/etc/system/local/.

For examples, see distsearch.conf.example. You must restart Splunk to enable configurations.

[harsmarvania57]

You need to create Folder structure like this on Deployer /opt/splunk/etc/apps/configuration/local/ and then copy distsearch.conf in that folder. After that push the bundle from deployer.

Thanks,
Harshil

[somesoni2]

1. Check if your search heads restarted after receiving new distsearch.conf.
2. If they did, check the status of search peers from Settings->Distributed Search->Search peers. The peers might be saying failed status due to authentication. Open each one of them and provide admin credentials.

[att35]

Thank you.

We checked the members and they did in fact need the credentials to be re-entered. Once that was done it was all set and each member was able to query the pool.