Deployment Architecture

How to distribute Distributed Search configuration using a deployer for a Search Head Cluster?

att35
Builder

Hi,

We recently set up a SH Cluster which includes 3 members and one deployer. Basic replication seems to be working fine(tested by creating a dashboard on one member), but running into issues when deploying configuration changes. What are the best practices when it comes to deploy a system configuration, e.g. distributed search peer's, from the Deployer to all the SH members?

If I understood the steps correctly, the only way to deploy anything from a deployer is to create an app under /opt/splunk/etc/shcluster/apps.
For this, I created a new folder called "configuration" and copied distsearch.conf from /opt/splunk/etc/system/local/distsearch.conf

Deployment was initiated using splunk apply shcluster-bundle. I can see the changes were accepted on the SH Member under /opt/splunk/etc/apps/configuration, but SH member is still unable to search any peer. Most likely these changes did not take effect.
Is this a wrong way to deploy any system changes using deployer?

Please advise.

Thanks,

~Abhi

0 Karma
1 Solution

jkat54
SplunkTrust
SplunkTrust

You actually configure each search head independently when it comes to distsearch.conf. You dont do it by app.

/opt/splunk/etc/system/local/distsearch.conf <--- will always take precedence over any distsearch.conf because it gets the highest priority

http://docs.splunk.com/Documentation/Splunk/6.2.0/Admin/Wheretofindtheconfigurationfiles

This means you MUST configure it on each search head independently and not via deployment app from the deployer:

http://docs.splunk.com/Documentation/Splunk/6.0/admin/Distsearchconf

To set custom configurations, place a distsearch.conf in $SPLUNK_HOME/etc/system/local/.

For examples, see distsearch.conf.example. You must restart Splunk to enable configurations.

View solution in original post

0 Karma

jkat54
SplunkTrust
SplunkTrust

You actually configure each search head independently when it comes to distsearch.conf. You dont do it by app.

/opt/splunk/etc/system/local/distsearch.conf <--- will always take precedence over any distsearch.conf because it gets the highest priority

http://docs.splunk.com/Documentation/Splunk/6.2.0/Admin/Wheretofindtheconfigurationfiles

This means you MUST configure it on each search head independently and not via deployment app from the deployer:

http://docs.splunk.com/Documentation/Splunk/6.0/admin/Distsearchconf

To set custom configurations, place a distsearch.conf in $SPLUNK_HOME/etc/system/local/.

For examples, see distsearch.conf.example. You must restart Splunk to enable configurations.

0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

You need to create Folder structure like this on Deployer /opt/splunk/etc/apps/configuration/local/ and then copy distsearch.conf in that folder. After that push the bundle from deployer.

Thanks,
Harshil

somesoni2
SplunkTrust
SplunkTrust
  1. Check if your search heads restarted after receiving new distsearch.conf.
  2. If they did, check the status of search peers from Settings->Distributed Search->Search peers. The peers might be saying failed status due to authentication. Open each one of them and provide admin credentials.
0 Karma

att35
Builder

Thank you.

We checked the members and they did in fact need the credentials to be re-entered. Once that was done it was all set and each member was able to query the pool.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...