Getting Data In

How to configure all nodes in a clustered environment to forward internal Splunk logs to the clustered indexers?

brent_weaver
Builder

I have a Splunk clustered environment built, both indexer and search head clustering. I would like to know how to make all internal Splunk logs go to the clustered indexers. Thanks!

1 Solution

supabuck
Path Finder

This can be accomplished by setting up monitoring statements:

[splunk@hostname default]$ pwd
/opt/splunk/etc/apps/SplunkLightForwarder/default
[splunk@hostname default]$ cat inputs.conf
#   Version 6.2.0
# these here just override and disable stuff that in system/default.

################################
# Data thru parsingQueue always
################################

[splunktcp]
route=has_key:tautology:parsingQueue;absent_key:tautology:parsingQueue

################################
# Make sure these get forwarded
################################

[monitor://$SPLUNK_HOME/var/log/splunk/splunkd.log]
_TCP_ROUTING = *
index = _internal

[monitor://$SPLUNK_HOME/var/log/splunk/metrics.log]
_TCP_ROUTING = *
index = _internal

It should then accept whatever you have in place for your outputs.conf in your case I would specify the ip addresses and ports for the indexers so that it is load balanced. You can set it up within that app folder in outputs.conf so that it only affects the data listed respective of the app context:

[tcpout:my-indexers]
autoLB = true
autoLBFrequency = 60
compressed = true
server = 10.0.0.4:9997,10.0.0.5:9997,10.0.0.6:9997

Basically with monitor stanzas on those files Splunk will be able to ingest it from my experience.

View solution in original post

supabuck
Path Finder

This can be accomplished by setting up monitoring statements:

[splunk@hostname default]$ pwd
/opt/splunk/etc/apps/SplunkLightForwarder/default
[splunk@hostname default]$ cat inputs.conf
#   Version 6.2.0
# these here just override and disable stuff that in system/default.

################################
# Data thru parsingQueue always
################################

[splunktcp]
route=has_key:tautology:parsingQueue;absent_key:tautology:parsingQueue

################################
# Make sure these get forwarded
################################

[monitor://$SPLUNK_HOME/var/log/splunk/splunkd.log]
_TCP_ROUTING = *
index = _internal

[monitor://$SPLUNK_HOME/var/log/splunk/metrics.log]
_TCP_ROUTING = *
index = _internal

It should then accept whatever you have in place for your outputs.conf in your case I would specify the ip addresses and ports for the indexers so that it is load balanced. You can set it up within that app folder in outputs.conf so that it only affects the data listed respective of the app context:

[tcpout:my-indexers]
autoLB = true
autoLBFrequency = 60
compressed = true
server = 10.0.0.4:9997,10.0.0.5:9997,10.0.0.6:9997

Basically with monitor stanzas on those files Splunk will be able to ingest it from my experience.

brent_weaver
Builder

Thank you all for your help. I just built a new index and sh cluster, I did nothing and one (of 3) of my search heads is logging to the cluster, why are the other two not?
why is this node different? Because I made it the sh cluster captin ?

0 Karma

brent_weaver
Builder

Thank you for the response! I did not know this but it totally makes sense. I did get it to work and if I am not mistaken I need to take different actions on the search head cluster nodes?!?! Is this not the case? So what I did with those machines is I used the shcluter deployer to deploy the internal log routing of splunk. This is the file:

# Turn off indexing on the search head
[indexAndForward]
index = false

[tcpout]
defaultGroup = Name
forwardedindex.filter.disable = true
indexAndForward = false

[tcpout:Name]
server=198.0.0.88:9997
autoLB = true

Is this not the best way to go?

Thanks!

0 Karma

supabuck
Path Finder

Hello Brent,

Given with what you are doing I believe that this configuration should work. According to https://docs.splunk.com/Documentation/Splunk/6.4.2/Admin/Outputsconf:

# If the [tcpout] stanza configures the indexAndForward attribute, the value
# of that attribute overrides the default value of "index". However, if you
# set "index" in the [indexAndForward] stanza, described below, it
# supersedes any value set in [tcpout].

[indexAndForward]
index = [true|false]
* If set to true, data is indexed.
* If set to false, data is not indexed.
* Default depends on whether the Splunk instance is configured as a
  forwarder, modified by any value configured for the indexAndForward
  attribute in [tcpout].

So, you probably would not need

indexAndForward = false

Within the outputs.conf [tcpout] configuration stanza.

0 Karma
Get Updates on the Splunk Community!

New Splunk Observability innovations: Deeper visibility and smarter alerting to ...

You asked, we delivered. Splunk Observability Cloud has several new innovations giving you deeper visibility ...

Synthetic Monitoring: Not your Grandma’s Polyester! Tech Talk: DevOps Edition

Register today and join TekStream on Tuesday, February 28 at 11am PT/2pm ET for a demonstration of Splunk ...

Instrumenting Java Websocket Messaging

Instrumenting Java Websocket MessagingThis article is a code-based discussion of passing OpenTelemetry trace ...