Deployment Architecture

How to distribute Distributed Search configuration using a deployer for a Search Head Cluster?

att35
Builder

Hi,

We recently set up a SH Cluster which includes 3 members and one deployer. Basic replication seems to be working fine(tested by creating a dashboard on one member), but running into issues when deploying configuration changes. What are the best practices when it comes to deploy a system configuration, e.g. distributed search peer's, from the Deployer to all the SH members?

If I understood the steps correctly, the only way to deploy anything from a deployer is to create an app under /opt/splunk/etc/shcluster/apps.
For this, I created a new folder called "configuration" and copied distsearch.conf from /opt/splunk/etc/system/local/distsearch.conf

Deployment was initiated using splunk apply shcluster-bundle. I can see the changes were accepted on the SH Member under /opt/splunk/etc/apps/configuration, but SH member is still unable to search any peer. Most likely these changes did not take effect.
Is this a wrong way to deploy any system changes using deployer?

Please advise.

Thanks,

~Abhi

0 Karma
1 Solution

jkat54
SplunkTrust
SplunkTrust

You actually configure each search head independently when it comes to distsearch.conf. You dont do it by app.

/opt/splunk/etc/system/local/distsearch.conf <--- will always take precedence over any distsearch.conf because it gets the highest priority

http://docs.splunk.com/Documentation/Splunk/6.2.0/Admin/Wheretofindtheconfigurationfiles

This means you MUST configure it on each search head independently and not via deployment app from the deployer:

http://docs.splunk.com/Documentation/Splunk/6.0/admin/Distsearchconf

To set custom configurations, place a distsearch.conf in $SPLUNK_HOME/etc/system/local/.

For examples, see distsearch.conf.example. You must restart Splunk to enable configurations.

View solution in original post

0 Karma

jkat54
SplunkTrust
SplunkTrust

You actually configure each search head independently when it comes to distsearch.conf. You dont do it by app.

/opt/splunk/etc/system/local/distsearch.conf <--- will always take precedence over any distsearch.conf because it gets the highest priority

http://docs.splunk.com/Documentation/Splunk/6.2.0/Admin/Wheretofindtheconfigurationfiles

This means you MUST configure it on each search head independently and not via deployment app from the deployer:

http://docs.splunk.com/Documentation/Splunk/6.0/admin/Distsearchconf

To set custom configurations, place a distsearch.conf in $SPLUNK_HOME/etc/system/local/.

For examples, see distsearch.conf.example. You must restart Splunk to enable configurations.

0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

You need to create Folder structure like this on Deployer /opt/splunk/etc/apps/configuration/local/ and then copy distsearch.conf in that folder. After that push the bundle from deployer.

Thanks,
Harshil

somesoni2
SplunkTrust
SplunkTrust
  1. Check if your search heads restarted after receiving new distsearch.conf.
  2. If they did, check the status of search peers from Settings->Distributed Search->Search peers. The peers might be saying failed status due to authentication. Open each one of them and provide admin credentials.
0 Karma

att35
Builder

Thank you.

We checked the members and they did in fact need the credentials to be re-entered. Once that was done it was all set and each member was able to query the pool.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...