Deployment Architecture

How can I forward the internal Splunk logs of a Splunk deployment to another Splunk

Ledio_Ago
Splunk Employee
Splunk Employee

Let's say I have a distributed Splunk environment, n indexers, one search head and a forwarder load balancing input data to these indexers. I like to pull out all of the internal Splunk logs from this deployment and have them forwarded to another Splunk for monitoring purposes. What's the best way of doing that?

2 Solutions

Lowell
Super Champion

That depends on the version. If you don't have any internal events being forwarded right now, you can enable forwarding of _internal events via the following:


Splunk 4.0 (and earlier):

You have to find the entry in inputs.conf that is responsible for monitoring your splunk internal log files and then add the mystical _TCP_ROUTING=* entry.

So add something like this to one of your `local/inputs.conf files:

[monitor://$SPLUNK_HOME/var/log/splunk]
_TCP_ROUTING = *

Note: Be aware that $SPLUNK_HOME/var/log/splunk/ log files can change based on platform. Also, in older version of splunk some of the log files were monitored individually, so you would have to add this for each [monitor:] entry.

Splunk 4.1

[tcpout]
forwardedindex.filter.disable = true

See the outputs.conf docs, specifically the section called Configuring which events are forwarded by index.


However, if you already have your _internal events forwarded and only want to forward your deployment log events, then you'll probably have to setup some transformers to re-route these events to a different index and then just forward that index. (That's just a guess. I hope, for your sake, it's not that complicated.... If so, we'll need some smarter people to step in.)

Hope this helps

View solution in original post

Ledio_Ago
Splunk Employee
Splunk Employee

With 4.1.2 I was able to do this in each machine that I wanted Splunk internal logs to be forwarded:

  • In outputs.conf

[tcpout]
forwardedindex.0.whitelist = _internal
forwardedindex.1.whitelist = _audit
forwardedindex.filter.disable = false
IndexAndForward = true
defaultGroup = invalidGroup

[tcpout:group1]
server = xxx.xxx.xxx.xxx:9997

  • in inputs.conf

[monitor://$SPLUNK_HOME/var/log/splunk]
host = index_1
_TCP_ROUTING=group1

[monitor://$SPLUNK_HOME/etc/splunk.version]
host = index_1
_TCP_ROUTING=group1

View solution in original post

Ledio_Ago
Splunk Employee
Splunk Employee

With 4.1.2 I was able to do this in each machine that I wanted Splunk internal logs to be forwarded:

  • In outputs.conf

[tcpout]
forwardedindex.0.whitelist = _internal
forwardedindex.1.whitelist = _audit
forwardedindex.filter.disable = false
IndexAndForward = true
defaultGroup = invalidGroup

[tcpout:group1]
server = xxx.xxx.xxx.xxx:9997

  • in inputs.conf

[monitor://$SPLUNK_HOME/var/log/splunk]
host = index_1
_TCP_ROUTING=group1

[monitor://$SPLUNK_HOME/etc/splunk.version]
host = index_1
_TCP_ROUTING=group1

View solution in original post

Lowell
Super Champion

That depends on the version. If you don't have any internal events being forwarded right now, you can enable forwarding of _internal events via the following:


Splunk 4.0 (and earlier):

You have to find the entry in inputs.conf that is responsible for monitoring your splunk internal log files and then add the mystical _TCP_ROUTING=* entry.

So add something like this to one of your `local/inputs.conf files:

[monitor://$SPLUNK_HOME/var/log/splunk]
_TCP_ROUTING = *

Note: Be aware that $SPLUNK_HOME/var/log/splunk/ log files can change based on platform. Also, in older version of splunk some of the log files were monitored individually, so you would have to add this for each [monitor:] entry.

Splunk 4.1

[tcpout]
forwardedindex.filter.disable = true

See the outputs.conf docs, specifically the section called Configuring which events are forwarded by index.


However, if you already have your _internal events forwarded and only want to forward your deployment log events, then you'll probably have to setup some transformers to re-route these events to a different index and then just forward that index. (That's just a guess. I hope, for your sake, it's not that complicated.... If so, we'll need some smarter people to step in.)

Hope this helps

View solution in original post

.conf21 CFS Extended through 5/20!

Don't miss your chance
to share your Splunk
wisdom in-person or
virtually at .conf21!

Call for Speakers has
been extended through
Thursday, 5/20!