How to configure all nodes in a clustered environment to forward internal Splunk logs to the clustered indexers?

[brent_weaver]

I have a Splunk clustered environment built, both indexer and search head clustering. I would like to know how to make all internal Splunk logs go to the clustered indexers. Thanks!

[supabuck · Accepted solution]

This can be accomplished by setting up monitoring statements:

[splunk@hostname default]$ pwd
/opt/splunk/etc/apps/SplunkLightForwarder/default
[splunk@hostname default]$ cat inputs.conf
#   Version 6.2.0
# these here just override and disable stuff that in system/default.

################################
# Data thru parsingQueue always
################################

[splunktcp]
route=has_key:tautology:parsingQueue;absent_key:tautology:parsingQueue

################################
# Make sure these get forwarded
################################

[monitor://$SPLUNK_HOME/var/log/splunk/splunkd.log]
_TCP_ROUTING = *
index = _internal

[monitor://$SPLUNK_HOME/var/log/splunk/metrics.log]
_TCP_ROUTING = *
index = _internal

It should then accept whatever you have in place for your outputs.conf in your case I would specify the ip addresses and ports for the indexers so that it is load balanced. You can set it up within that app folder in outputs.conf so that it only affects the data listed respective of the app context:

[tcpout:my-indexers]
autoLB = true
autoLBFrequency = 60
compressed = true
server = 10.0.0.4:9997,10.0.0.5:9997,10.0.0.6:9997

Basically with monitor stanzas on those files Splunk will be able to ingest it from my experience.

[brent_weaver]

Thank you all for your help. I just built a new index and sh cluster, I did nothing and one (of 3) of my search heads is logging to the cluster, why are the other two not?
why is this node different? Because I made it the sh cluster captin ?

[brent_weaver]

Thank you for the response! I did not know this but it totally makes sense. I did get it to work and if I am not mistaken I need to take different actions on the search head cluster nodes?!?! Is this not the case? So what I did with those machines is I used the shcluter deployer to deploy the internal log routing of splunk. This is the file:

# Turn off indexing on the search head
[indexAndForward]
index = false

[tcpout]
defaultGroup = Name
forwardedindex.filter.disable = true
indexAndForward = false

[tcpout:Name]
server=198.0.0.88:9997
autoLB = true

Is this not the best way to go?

Thanks!

[supabuck]

Hello Brent,

Given with what you are doing I believe that this configuration should work. According to https://docs.splunk.com/Documentation/Splunk/6.4.2/Admin/Outputsconf:

# If the [tcpout] stanza configures the indexAndForward attribute, the value
# of that attribute overrides the default value of "index". However, if you
# set "index" in the [indexAndForward] stanza, described below, it
# supersedes any value set in [tcpout].

[indexAndForward]
index = [true|false]
* If set to true, data is indexed.
* If set to false, data is not indexed.
* Default depends on whether the Splunk instance is configured as a
  forwarder, modified by any value configured for the indexAndForward
  attribute in [tcpout].

So, you probably would not need

indexAndForward = false

Within the outputs.conf [tcpout] configuration stanza.