We have a setup of 6 search head clusters connected to 2 multi-site indexer clusters, using Splunk v6.3.1. My objective is to forward search head cluster data to multi-site indexer cluster.
As per following documentation, I understand this can be achieved using outputs.conf
tcpoutput stanza (server=host1, host2, host3)
But I want the cluster master node to handle this (similar to indexer discovery in forwarders).
q1. Is this achievable?
q2. Will the data be replicated if it is sent to a set of indexers in multi-site indexers?
Yes. Technically possible. We need to add the role of forwarder to search head. We also need to disable indexing on the SH so data is not duplicated according to the following section.
“2. Configure the search head as a forwarder. Create an outputs.conf file on the search head that configures the search head for load-balanced forwarding across the set of search peers (indexers). You must also turn off indexing on the search head, so that the search head does not both retain the data locally as well as forward it to the search peers.”
Since the SH now is also a forwarder, we should be able to use the outputs.conf like, on any other forwarder, so indexerDiscovery should work the same. Forwarder contacts the cluster master and gets the list of indexers.
We have tested this in our LAB and implemented in PROD.
I've copied my outputs.conf from my other forwarders on my search head in /opt/splunk/etc/system/local/outputs.conf :
[indexer_discovery:master1] pass4SymmKey = ************ master_uri = https://*********:8089 [tcpout:group1] autoLBFrequency = 30 forceTimebasedAutoLB = true indexerDiscovery = master1 useACK = true [tcpout] defaultGroup = group1
After a restart I'm able to add monitors to the indexes present on the peers which seems to mean that discovery occured but I see no data from the heads when I do a search and "list forward-server" on the search heads returns nothing
root@********-spk-search1:/opt/splunk/etc# /opt/splunk/bin/splunk list forward-server Active forwards: None Configured but inactive forwards: None
Did I miss something ?
Also how do I disable local indexing on the search heads ?