Deployment Architecture

How can I forward the internal Splunk logs of a Splunk deployment to another Splunk

Ledio_Ago
Splunk Employee
Splunk Employee

Let's say I have a distributed Splunk environment, n indexers, one search head and a forwarder load balancing input data to these indexers. I like to pull out all of the internal Splunk logs from this deployment and have them forwarded to another Splunk for monitoring purposes. What's the best way of doing that?

2 Solutions

Lowell
Super Champion

That depends on the version. If you don't have any internal events being forwarded right now, you can enable forwarding of _internal events via the following:


Splunk 4.0 (and earlier):

You have to find the entry in inputs.conf that is responsible for monitoring your splunk internal log files and then add the mystical _TCP_ROUTING=* entry.

So add something like this to one of your `local/inputs.conf files:

[monitor://$SPLUNK_HOME/var/log/splunk]
_TCP_ROUTING = *

Note: Be aware that $SPLUNK_HOME/var/log/splunk/ log files can change based on platform. Also, in older version of splunk some of the log files were monitored individually, so you would have to add this for each [monitor:] entry.

Splunk 4.1

[tcpout]
forwardedindex.filter.disable = true

See the outputs.conf docs, specifically the section called Configuring which events are forwarded by index.


However, if you already have your _internal events forwarded and only want to forward your deployment log events, then you'll probably have to setup some transformers to re-route these events to a different index and then just forward that index. (That's just a guess. I hope, for your sake, it's not that complicated.... If so, we'll need some smarter people to step in.)

Hope this helps

View solution in original post

Ledio_Ago
Splunk Employee
Splunk Employee

With 4.1.2 I was able to do this in each machine that I wanted Splunk internal logs to be forwarded:

  • In outputs.conf

[tcpout]
forwardedindex.0.whitelist = _internal
forwardedindex.1.whitelist = _audit
forwardedindex.filter.disable = false
IndexAndForward = true
defaultGroup = invalidGroup

[tcpout:group1]
server = xxx.xxx.xxx.xxx:9997

  • in inputs.conf

[monitor://$SPLUNK_HOME/var/log/splunk]
host = index_1
_TCP_ROUTING=group1

[monitor://$SPLUNK_HOME/etc/splunk.version]
host = index_1
_TCP_ROUTING=group1

View solution in original post

Ledio_Ago
Splunk Employee
Splunk Employee

With 4.1.2 I was able to do this in each machine that I wanted Splunk internal logs to be forwarded:

  • In outputs.conf

[tcpout]
forwardedindex.0.whitelist = _internal
forwardedindex.1.whitelist = _audit
forwardedindex.filter.disable = false
IndexAndForward = true
defaultGroup = invalidGroup

[tcpout:group1]
server = xxx.xxx.xxx.xxx:9997

  • in inputs.conf

[monitor://$SPLUNK_HOME/var/log/splunk]
host = index_1
_TCP_ROUTING=group1

[monitor://$SPLUNK_HOME/etc/splunk.version]
host = index_1
_TCP_ROUTING=group1

Lowell
Super Champion

That depends on the version. If you don't have any internal events being forwarded right now, you can enable forwarding of _internal events via the following:


Splunk 4.0 (and earlier):

You have to find the entry in inputs.conf that is responsible for monitoring your splunk internal log files and then add the mystical _TCP_ROUTING=* entry.

So add something like this to one of your `local/inputs.conf files:

[monitor://$SPLUNK_HOME/var/log/splunk]
_TCP_ROUTING = *

Note: Be aware that $SPLUNK_HOME/var/log/splunk/ log files can change based on platform. Also, in older version of splunk some of the log files were monitored individually, so you would have to add this for each [monitor:] entry.

Splunk 4.1

[tcpout]
forwardedindex.filter.disable = true

See the outputs.conf docs, specifically the section called Configuring which events are forwarded by index.


However, if you already have your _internal events forwarded and only want to forward your deployment log events, then you'll probably have to setup some transformers to re-route these events to a different index and then just forward that index. (That's just a guess. I hope, for your sake, it's not that complicated.... If so, we'll need some smarter people to step in.)

Hope this helps

Get Updates on the Splunk Community!

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...

Splunkbase | Splunk Dashboard Examples App for SimpleXML End of Life

The Splunk Dashboard Examples App for SimpleXML will reach end of support on Dec 19, 2024, after which no new ...

Understanding Generative AI Techniques and Their Application in Cybersecurity

Watch On-Demand Artificial intelligence is the talk of the town nowadays, with industries of all kinds ...