Defining search-peer with configuration only

[beaumaris]

We are trying to configure search peers with search heads and indexers using configuration files only. Splunk provides a command line that describes the indexer and authorization. For example:

./splunk add search-server --host :8089 --auth admin: -remoteUsername admin -remotePassword

During the operation of that command on the Search Head, Splunk defines the search peer in etc/system/local/distsearch.conf and also sends the trusted.pem file to the indexer in the directory $SPLUNK_HOME/etc/auth/distServerKeys//trusted.pem

We are looking for a configuration-only solution to setting up search peers without using the CLI command. We are not using auto-discovery since our customer installations will not allow multicast. Is there any way to perform the token exchange with configuration files? Is there a web services call that can perform the token-passing?

[thambisetty]

https://docs.splunk.com/Documentation/Splunk/7.3.1/DistSearch/Configuredistributedsearch#Edit_distse...

[triest]

I realize this question is a bit old, but I came across it so I thought I'd try to answer it in hopes of helping others.

To setup search peers by only modifying files on the file system (oaky a restart of Splunk is probably also required), there are two basic steps:

1. Configure distsearch.conf
2. Copy the certificate from the search head to the indexers.

Step 1 is how the search heads know where to send search jobs while step 2 is how the indexers know the search is authenticated -- this is critical since the search heads are the ones that enforce permissions.

Step 1. Configuring distsearch.conf

In distsearch.conf there are two relevant settings:
<dl>
<dt>servers</dt>
<dd>a comma separated list of search peers</dd>
<dd>servers = index1.example.com:8089,index2.example.com:8089</dd>
<dt>disabled_servers</dt>
<dd>a comma separated list of search peers, just like servers, except these search peers are in a disabled state</dd>
<dd>disabled_servers = index4.examples.com:8089,index5.example.com</dd>
</dl>

In typical Splunk fashion, if you edit the search peers from the web interface they settings are stored in $SPLUNK_HOME/etc/system/local but you can put them in an app $SPLUNK_HOME/etc/apps/search_peers/local so that its easy to push them from a deployment server.

Step 2 Copy the certificate from the search heads to the search peers

On the search head, grab a copy of $SPLUNK_HOME/etc/auth/distServerKeys/trusted.pem and put it in $SPLUNK_HOME/etc/auth/distServerKeys/${search_head_host_name}/trusted.pem (e.g. searchhead1 go to index1 and save the file as $SPLUNK_HOME/etc/auth/distServerKeys/searchhead1/trusted.pem)

Step 3: Restart

After making the change to distsearch.conf, you'll need to restart. After adding the search head public keys, I would restart although I'm not positive its actually required.

Since the trusted.pem is a public key, it should be safe to distribute it via a configuration management system (e.g. Puppet, Ansibule, etc). I'm not aware of an easy way to distribute the keys via the deployment server. Since they only
need to be on the search peers (typically indexers, although for your Splunk instance that runs deployment console, you'll want your search heads to be peers), the set of servers that need the keys is likely within your control; this means I can use Puppet to distribute them even though we highly depend on the Deployment Server for general configuration management when it comes to Splunk.

[samlll42]

We have the exact same problem (i.e adding search peers through a Chef Recipe that searches for all the indexers dynamically and populates the distsearch.conf) the server is added but authentication fails (probably because editing distsearch.conf is not enough).

So how can we add search peers entirely through command files without running the command line tool? (I took a look at the REST end points above and I don't see anything relevant to add new peers - Did I miss it?)

[Jeremiah]

I've been trying to do the same thing with puppet, have you had any luck?

[triest]

@Jeremiah Did you get it working? If not, I used Puppet with Vagrant to create virtual environments over a year ago and earlier today I used Puppet to setup new Indexers so I have done it and would be glad to try and help.

[Damien_Dallimor]

Have a look at these REST Endpoints :

http://docs.splunk.com/Documentation/Splunk/latest/RESTAPI/RESTdeploy#search.2Fdistributed.2Fpeers

http://docs.splunk.com/Documentation/Splunk/latest/RESTAPI/RESTdeploy#search.2Fdistributed.2Fpeers.2...