Can I use indexer discovery to forward search head cluster data to multi-site indexer cluster?

[basanthp]

We have a setup of 6 search head clusters connected to 2 multi-site indexer clusters, using Splunk v6.3.1. My objective is to forward search head cluster data to multi-site indexer cluster.

As per following documentation, I understand this can be achieved using outputs.conf
http://docs.splunk.com/Documentation/Splunk/6.3.1/DistSearch/Forwardsearchheaddata
tcpoutput stanza (server=host1, host2, host3)

But I want the cluster master node to handle this (similar to indexer discovery in forwarders).

q1. Is this achievable?
q2. Will the data be replicated if it is sent to a set of indexers in multi-site indexers?

[basanthp · Accepted solution]

Yes. Technically possible. We need to add the role of forwarder to search head. We also need to disable indexing on the SH so data is not duplicated according to the following section.

“2. Configure the search head as a forwarder. Create an outputs.conf file on the search head that configures the search head for load-balanced forwarding across the set of search peers (indexers). You must also turn off indexing on the search head, so that the search head does not both retain the data locally as well as forward it to the search peers.”

Since the SH now is also a forwarder, we should be able to use the outputs.conf like, on any other forwarder, so indexerDiscovery should work the same. Forwarder contacts the cluster master and gets the list of indexers.

We have tested this in our LAB and implemented in PROD.

[sylvainlectra]

Hi,

I've copied my outputs.conf from my other forwarders on my search head in /opt/splunk/etc/system/local/outputs.conf :

[indexer_discovery:master1]
pass4SymmKey = ************
master_uri = https://*********:8089

[tcpout:group1]
autoLBFrequency = 30
forceTimebasedAutoLB = true
indexerDiscovery = master1
useACK = true

[tcpout]
defaultGroup = group1

After a restart I'm able to add monitors to the indexes present on the peers which seems to mean that discovery occured but I see no data from the heads when I do a search and "list forward-server" on the search heads returns nothing

root@********-spk-search1:/opt/splunk/etc# /opt/splunk/bin/splunk list forward-server
Active forwards:
        None
Configured but inactive forwards:
        None

Did I miss something ?

Also how do I disable local indexing on the search heads ?

Regards.

[srajarat2]

https://docs.splunk.com/Documentation/Splunk/7.3.2/DistSearch/Forwardsearchheaddata