Getting Data In

Should I configure a universal forwarder to forward data to the master node in an indexer cluster?

New Member

Setting up a Splunk indexer cluster consists of the following:

idx01 : indexer mode: master
idx02 : indexer mode: slave
idx03 : indexer mode: slave
idx04 : indexer mode: slave
sh01 : search head
sh02 : search head
sh03 : search head
uf01 : universal forwarder
uf02 : universal forwarder

# cat ../etc/system/local/outputs.conf

[tcpout]
defaultGroup = default-autolb-group
[tcpout-server://idx01:9997]
[tcpout-server://idx02:9997]
[tcpout-server://idx03:9997]
[tcpout-server://idx04:9997]
[tcpout:default-autolb-group]
disabled = false
server = idx01:9997,idx02:9997,idx03:9997,idx04:9997

# ./splunk list forward-server

Active forwards:
        idx02:9997
Configured but inactive forwards:
        idx01:9997
        idx03:9997
        idx04:9997

What is the best practice here?

  1. Should 'universal forwarder' forward to idx01 (master-node) only and let master-node do the auto-load-balancing to all slave indexers?

  2. Should 'universal forwarder' forward to all indexers (idx01, idx02, idx03, idx04) and let the universal forwarder does the auto-load-balancing?

  3. Should 'universal forwarder' NOT forward to master-node(idx01), but to other slave indexers (idx02, idx03, idx04) and let the universal forwarder do the auto-load-balancing?

or something else?

Thank you.

0 Karma
1 Solution

SplunkTrust
SplunkTrust

The recommended method is using Indexer Discovery using the Cluster Master nodes. See following link for steps and benefit of Indexer discovery method over traditional method of directly connecting to peer nodes (both works just fine).

http://docs.splunk.com/Documentation/Splunk/6.5.0/Indexer/useforwarders

http://docs.splunk.com/Documentation/Splunk/6.5.0/Indexer/indexerdiscovery

View solution in original post

SplunkTrust
SplunkTrust

How did you get the idea of forwarding data to the master node in the first place???

And yeah, somesoni2 is right.

0 Karma

New Member

From the following link says
http://docs.splunk.com/Documentation/Splunk/latest/Indexer/Forwardmasterdata

"Configure the master as a forwarder. Create an outputs.conf file on the master node that configures it for load-balanced forwarding across the set of peer nodes. You must also turn off indexing on the master, so that the master does not both retain the data locally as well as forward it to the peers."

Does this mean option 1 is the best practice in this situation?

0 Karma

SplunkTrust
SplunkTrust

The above link is for something else. It's sending _internal (and anything else being monitored) of the master node to indexers, instead of storing it locally. The link in my answer is for setting up forwarders to use/forwarder data to indexer cluster.

New Member

@somesoni2,

After reading steps mentioned in links provided by you,
I added stanza 'indexer_discovery' to the bottom part of
server.conf on idx01 (indexer-master) and outputs.conf (universal forwarder).

I did 'splunk restart' on idx01 and uf01.

However, I see consistent ERROR on universal forwarder's log.
No Error on Indexer-master's log (splunkd.log).

Is that ERROR on unvirsal forwarder's a legit error or can be ignored?

INDEXER-MASTER
--------------
# cat server.conf
[general]
serverName = idx01
pass4SymmKey = $1$oNHXGCjyfX/G
site = default

[sslConfig]
sslPassword = $1$952DRG20Ni3G

[lmpool:auto_generated_pool_download-trial]
description = auto_generated_pool_download-trial
quota = MAX
slaves = *
stack_id = download-trial

[lmpool:auto_generated_pool_forwarder]
description = auto_generated_pool_forwarder
quota = MAX
slaves = *
stack_id = forwarder

[lmpool:auto_generated_pool_free]
description = auto_generated_pool_free
quota = MAX
slaves = *
stack_id = free

[clustering]
access_logging_for_heartbeats = 1
cluster_label = qpsplunk
max_peer_build_load = 5
mode = master
pass4SymmKey = $1$ssndEzbb
service_interval = 1

[indexer_discovery]
pass4SymmKey = $1$ssnFEyzldW7G
polling_rate = 10
indexerWeightByDiskCapacity = FALSE



UNIFERSAL FORWARDER
-------------------
# cat outputs.conf
[indexer_discovery:qpsplunkdiscovery]
pass4SymmKey = $1$M05xrAKiR/Vn
master_uri = https://idx01:8089

[tcpout:group1]
autoLBFrequencey = 30
forceTimebasedAutoLB = true
indexerDiscovery = qpsplunkdiscovery
useACK = true

[tcpout]
defaultGroup = group1






UNIVERSAL FORWARDER
-------------------
[root@uf01 0 /opt/splunkforwarder/bin]# less ../var/log/splunk/splunkd.log

10-18-2016 20:16:04.997 +0000 ERROR IndexerDiscoveryHeartbeatThread - failed to parse response payload for group=group1, err=failed to extract FwdTarget from json node={"hostport":"?","ssl":false,"indexing_disk_space":-1}http_response=OK
10-18-2016 20:16:10.006 +0000 ERROR IndexerDiscoveryHeartbeatThread - failed to parse response payload for group=group1, err=failed to extract FwdTarget from json node={"hostport":"?","ssl":false,"indexing_disk_space":-1}http_response=OK




INDEXER
-------
[root@idx01 0 /opt/splunk/bin]# less ../var/log/splunk/splunkd.log

10-18-2016 20:07:58.941 +0000 INFO  CMIndexerDiscovery - Request rate limiting = 10 requests per second
10-18-2016 20:07:58.941 +0000 INFO  CMIndexerDiscovery - Indexer weight by disk capacity = 0
10-18-2016 20:07:58.941 +0000 INFO  CMIndexerDiscovery - Registering new forwarder 5AB0AD22-839F-4340-89EC-7CCC6E3C1F8F (total: 1). Heartbeat assigned for next check: 30 seconds
10-18-2016 20:08:23.655 +0000 WARN  DistributedBundleReplicationManager - Asynchronous bundle replication to 3 peer(s) succeeded; however it took too long (longer than 10 seconds): elapsed_ms=19134, tar_elapsed_ms=3751, bundle_file_size=76000KB, replication_id=1476821284, replication_reason="async replication allowed"
0 Karma

New Member

I reinstalled from scratch, no more ERROR IndexerDiscoveryHeartbeatThread.
It is working now.

Thanks you.

0 Karma

SplunkTrust
SplunkTrust

The recommended method is using Indexer Discovery using the Cluster Master nodes. See following link for steps and benefit of Indexer discovery method over traditional method of directly connecting to peer nodes (both works just fine).

http://docs.splunk.com/Documentation/Splunk/6.5.0/Indexer/useforwarders

http://docs.splunk.com/Documentation/Splunk/6.5.0/Indexer/indexerdiscovery

View solution in original post