Getting Data In

How to configure all nodes in a clustered environment to forward internal Splunk logs to the clustered indexers?

brent_weaver
Builder

I have a Splunk clustered environment built, both indexer and search head clustering. I would like to know how to make all internal Splunk logs go to the clustered indexers. Thanks!

1 Solution

supabuck
Path Finder

This can be accomplished by setting up monitoring statements:

[splunk@hostname default]$ pwd
/opt/splunk/etc/apps/SplunkLightForwarder/default
[splunk@hostname default]$ cat inputs.conf
#   Version 6.2.0
# these here just override and disable stuff that in system/default.

################################
# Data thru parsingQueue always
################################

[splunktcp]
route=has_key:tautology:parsingQueue;absent_key:tautology:parsingQueue

################################
# Make sure these get forwarded
################################

[monitor://$SPLUNK_HOME/var/log/splunk/splunkd.log]
_TCP_ROUTING = *
index = _internal

[monitor://$SPLUNK_HOME/var/log/splunk/metrics.log]
_TCP_ROUTING = *
index = _internal

It should then accept whatever you have in place for your outputs.conf in your case I would specify the ip addresses and ports for the indexers so that it is load balanced. You can set it up within that app folder in outputs.conf so that it only affects the data listed respective of the app context:

[tcpout:my-indexers]
autoLB = true
autoLBFrequency = 60
compressed = true
server = 10.0.0.4:9997,10.0.0.5:9997,10.0.0.6:9997

Basically with monitor stanzas on those files Splunk will be able to ingest it from my experience.

View solution in original post

supabuck
Path Finder

This can be accomplished by setting up monitoring statements:

[splunk@hostname default]$ pwd
/opt/splunk/etc/apps/SplunkLightForwarder/default
[splunk@hostname default]$ cat inputs.conf
#   Version 6.2.0
# these here just override and disable stuff that in system/default.

################################
# Data thru parsingQueue always
################################

[splunktcp]
route=has_key:tautology:parsingQueue;absent_key:tautology:parsingQueue

################################
# Make sure these get forwarded
################################

[monitor://$SPLUNK_HOME/var/log/splunk/splunkd.log]
_TCP_ROUTING = *
index = _internal

[monitor://$SPLUNK_HOME/var/log/splunk/metrics.log]
_TCP_ROUTING = *
index = _internal

It should then accept whatever you have in place for your outputs.conf in your case I would specify the ip addresses and ports for the indexers so that it is load balanced. You can set it up within that app folder in outputs.conf so that it only affects the data listed respective of the app context:

[tcpout:my-indexers]
autoLB = true
autoLBFrequency = 60
compressed = true
server = 10.0.0.4:9997,10.0.0.5:9997,10.0.0.6:9997

Basically with monitor stanzas on those files Splunk will be able to ingest it from my experience.

brent_weaver
Builder

Thank you all for your help. I just built a new index and sh cluster, I did nothing and one (of 3) of my search heads is logging to the cluster, why are the other two not?
why is this node different? Because I made it the sh cluster captin ?

0 Karma

brent_weaver
Builder

Thank you for the response! I did not know this but it totally makes sense. I did get it to work and if I am not mistaken I need to take different actions on the search head cluster nodes?!?! Is this not the case? So what I did with those machines is I used the shcluter deployer to deploy the internal log routing of splunk. This is the file:

# Turn off indexing on the search head
[indexAndForward]
index = false

[tcpout]
defaultGroup = Name
forwardedindex.filter.disable = true
indexAndForward = false

[tcpout:Name]
server=198.0.0.88:9997
autoLB = true

Is this not the best way to go?

Thanks!

0 Karma

supabuck
Path Finder

Hello Brent,

Given with what you are doing I believe that this configuration should work. According to https://docs.splunk.com/Documentation/Splunk/6.4.2/Admin/Outputsconf:

# If the [tcpout] stanza configures the indexAndForward attribute, the value
# of that attribute overrides the default value of "index". However, if you
# set "index" in the [indexAndForward] stanza, described below, it
# supersedes any value set in [tcpout].

[indexAndForward]
index = [true|false]
* If set to true, data is indexed.
* If set to false, data is not indexed.
* Default depends on whether the Splunk instance is configured as a
  forwarder, modified by any value configured for the indexAndForward
  attribute in [tcpout].

So, you probably would not need

indexAndForward = false

Within the outputs.conf [tcpout] configuration stanza.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...