Deployment Architecture

How to distribute Distributed Search configuration using a deployer for a Search Head Cluster?

Contributor

Hi,

We recently set up a SH Cluster which includes 3 members and one deployer. Basic replication seems to be working fine(tested by creating a dashboard on one member), but running into issues when deploying configuration changes. What are the best practices when it comes to deploy a system configuration, e.g. distributed search peer's, from the Deployer to all the SH members?

If I understood the steps correctly, the only way to deploy anything from a deployer is to create an app under /opt/splunk/etc/shcluster/apps.
For this, I created a new folder called "configuration" and copied distsearch.conf from /opt/splunk/etc/system/local/distsearch.conf

Deployment was initiated using splunk apply shcluster-bundle. I can see the changes were accepted on the SH Member under /opt/splunk/etc/apps/configuration, but SH member is still unable to search any peer. Most likely these changes did not take effect.
Is this a wrong way to deploy any system changes using deployer?

Please advise.

Thanks,

~Abhi

0 Karma
1 Solution

SplunkTrust
SplunkTrust

You actually configure each search head independently when it comes to distsearch.conf. You dont do it by app.

/opt/splunk/etc/system/local/distsearch.conf <--- will always take precedence over any distsearch.conf because it gets the highest priority

http://docs.splunk.com/Documentation/Splunk/6.2.0/Admin/Wheretofindtheconfigurationfiles

This means you MUST configure it on each search head independently and not via deployment app from the deployer:

http://docs.splunk.com/Documentation/Splunk/6.0/admin/Distsearchconf

To set custom configurations, place a distsearch.conf in $SPLUNK_HOME/etc/system/local/.

For examples, see distsearch.conf.example. You must restart Splunk to enable configurations.

View solution in original post

0 Karma

SplunkTrust
SplunkTrust

You actually configure each search head independently when it comes to distsearch.conf. You dont do it by app.

/opt/splunk/etc/system/local/distsearch.conf <--- will always take precedence over any distsearch.conf because it gets the highest priority

http://docs.splunk.com/Documentation/Splunk/6.2.0/Admin/Wheretofindtheconfigurationfiles

This means you MUST configure it on each search head independently and not via deployment app from the deployer:

http://docs.splunk.com/Documentation/Splunk/6.0/admin/Distsearchconf

To set custom configurations, place a distsearch.conf in $SPLUNK_HOME/etc/system/local/.

For examples, see distsearch.conf.example. You must restart Splunk to enable configurations.

View solution in original post

0 Karma

SplunkTrust
SplunkTrust

You need to create Folder structure like this on Deployer /opt/splunk/etc/apps/configuration/local/ and then copy distsearch.conf in that folder. After that push the bundle from deployer.

Thanks,
Harshil

SplunkTrust
SplunkTrust
  1. Check if your search heads restarted after receiving new distsearch.conf.
  2. If they did, check the status of search peers from Settings->Distributed Search->Search peers. The peers might be saying failed status due to authentication. Open each one of them and provide admin credentials.
0 Karma

Contributor

Thank you.

We checked the members and they did in fact need the credentials to be re-entered. Once that was done it was all set and each member was able to query the pool.

0 Karma