Deployment Architecture

How can I forward the internal Splunk logs of a Splunk deployment to another Splunk

Ledio_Ago
Splunk Employee
Splunk Employee

Let's say I have a distributed Splunk environment, n indexers, one search head and a forwarder load balancing input data to these indexers. I like to pull out all of the internal Splunk logs from this deployment and have them forwarded to another Splunk for monitoring purposes. What's the best way of doing that?

2 Solutions

Lowell
Super Champion

That depends on the version. If you don't have any internal events being forwarded right now, you can enable forwarding of _internal events via the following:


Splunk 4.0 (and earlier):

You have to find the entry in inputs.conf that is responsible for monitoring your splunk internal log files and then add the mystical _TCP_ROUTING=* entry.

So add something like this to one of your `local/inputs.conf files:

[monitor://$SPLUNK_HOME/var/log/splunk]
_TCP_ROUTING = *

Note: Be aware that $SPLUNK_HOME/var/log/splunk/ log files can change based on platform. Also, in older version of splunk some of the log files were monitored individually, so you would have to add this for each [monitor:] entry.

Splunk 4.1

[tcpout]
forwardedindex.filter.disable = true

See the outputs.conf docs, specifically the section called Configuring which events are forwarded by index.


However, if you already have your _internal events forwarded and only want to forward your deployment log events, then you'll probably have to setup some transformers to re-route these events to a different index and then just forward that index. (That's just a guess. I hope, for your sake, it's not that complicated.... If so, we'll need some smarter people to step in.)

Hope this helps

View solution in original post

Ledio_Ago
Splunk Employee
Splunk Employee

With 4.1.2 I was able to do this in each machine that I wanted Splunk internal logs to be forwarded:

  • In outputs.conf

[tcpout]
forwardedindex.0.whitelist = _internal
forwardedindex.1.whitelist = _audit
forwardedindex.filter.disable = false
IndexAndForward = true
defaultGroup = invalidGroup

[tcpout:group1]
server = xxx.xxx.xxx.xxx:9997

  • in inputs.conf

[monitor://$SPLUNK_HOME/var/log/splunk]
host = index_1
_TCP_ROUTING=group1

[monitor://$SPLUNK_HOME/etc/splunk.version]
host = index_1
_TCP_ROUTING=group1

View solution in original post

Ledio_Ago
Splunk Employee
Splunk Employee

With 4.1.2 I was able to do this in each machine that I wanted Splunk internal logs to be forwarded:

  • In outputs.conf

[tcpout]
forwardedindex.0.whitelist = _internal
forwardedindex.1.whitelist = _audit
forwardedindex.filter.disable = false
IndexAndForward = true
defaultGroup = invalidGroup

[tcpout:group1]
server = xxx.xxx.xxx.xxx:9997

  • in inputs.conf

[monitor://$SPLUNK_HOME/var/log/splunk]
host = index_1
_TCP_ROUTING=group1

[monitor://$SPLUNK_HOME/etc/splunk.version]
host = index_1
_TCP_ROUTING=group1

View solution in original post

Lowell
Super Champion

That depends on the version. If you don't have any internal events being forwarded right now, you can enable forwarding of _internal events via the following:


Splunk 4.0 (and earlier):

You have to find the entry in inputs.conf that is responsible for monitoring your splunk internal log files and then add the mystical _TCP_ROUTING=* entry.

So add something like this to one of your `local/inputs.conf files:

[monitor://$SPLUNK_HOME/var/log/splunk]
_TCP_ROUTING = *

Note: Be aware that $SPLUNK_HOME/var/log/splunk/ log files can change based on platform. Also, in older version of splunk some of the log files were monitored individually, so you would have to add this for each [monitor:] entry.

Splunk 4.1

[tcpout]
forwardedindex.filter.disable = true

See the outputs.conf docs, specifically the section called Configuring which events are forwarded by index.


However, if you already have your _internal events forwarded and only want to forward your deployment log events, then you'll probably have to setup some transformers to re-route these events to a different index and then just forward that index. (That's just a guess. I hope, for your sake, it's not that complicated.... If so, we'll need some smarter people to step in.)

Hope this helps

View solution in original post

.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!