How can I forward the internal Splunk logs of a Splunk deployment to another Splunk

[Ledio_Ago]

Let's say I have a distributed Splunk environment, n indexers, one search head and a forwarder load balancing input data to these indexers. I like to pull out all of the internal Splunk logs from this deployment and have them forwarded to another Splunk for monitoring purposes. What's the best way of doing that?

[Lowell · Accepted solution]

That depends on the version. If you don't have any internal events being forwarded right now, you can enable forwarding of _internal events via the following:

---

Splunk 4.0 (and earlier):

You have to find the entry in inputs.conf that is responsible for monitoring your splunk internal log files and then add the mystical _TCP_ROUTING=* entry.

So add something like this to one of your `local/inputs.conf files:

[monitor://$SPLUNK_HOME/var/log/splunk]
_TCP_ROUTING = *

Note: Be aware that $SPLUNK_HOME/var/log/splunk/ log files can change based on platform. Also, in older version of splunk some of the log files were monitored individually, so you would have to add this for each [monitor:] entry.

Splunk 4.1

[tcpout]
forwardedindex.filter.disable = true

See the outputs.conf docs, specifically the section called Configuring which events are forwarded by index.

---

However, if you already have your _internal events forwarded and only want to forward your deployment log events, then you'll probably have to setup some transformers to re-route these events to a different index and then just forward that index. (That's just a guess. I hope, for your sake, it's not that complicated.... If so, we'll need some smarter people to step in.)

Hope this helps

[Ledio_Ago · Accepted solution]

With 4.1.2 I was able to do this in each machine that I wanted Splunk internal logs to be forwarded:

• In outputs.conf

[tcpout]
forwardedindex.0.whitelist = _internal
forwardedindex.1.whitelist = _audit
forwardedindex.filter.disable = false
IndexAndForward = true
defaultGroup = invalidGroup

[tcpout:group1]
server = xxx.xxx.xxx.xxx:9997

• in inputs.conf

[monitor://$SPLUNK_HOME/var/log/splunk]
host = index_1
_TCP_ROUTING=group1

[monitor://$SPLUNK_HOME/etc/splunk.version]
host = index_1
_TCP_ROUTING=group1

[Lowell · Accepted solution]

That depends on the version. If you don't have any internal events being forwarded right now, you can enable forwarding of _internal events via the following:

---

Splunk 4.0 (and earlier):

You have to find the entry in inputs.conf that is responsible for monitoring your splunk internal log files and then add the mystical _TCP_ROUTING=* entry.

So add something like this to one of your `local/inputs.conf files:

[monitor://$SPLUNK_HOME/var/log/splunk]
_TCP_ROUTING = *

Note: Be aware that $SPLUNK_HOME/var/log/splunk/ log files can change based on platform. Also, in older version of splunk some of the log files were monitored individually, so you would have to add this for each [monitor:] entry.

Splunk 4.1

[tcpout]
forwardedindex.filter.disable = true

See the outputs.conf docs, specifically the section called Configuring which events are forwarded by index.

---

However, if you already have your _internal events forwarded and only want to forward your deployment log events, then you'll probably have to setup some transformers to re-route these events to a different index and then just forward that index. (That's just a guess. I hope, for your sake, it's not that complicated.... If so, we'll need some smarter people to step in.)

Hope this helps