How to configure heavy forwarder _internal index forwarding?

[mike_k]

I have a single instance Splunk Enterprise deployment running on Linux. I have a bunch of data feeding into my indexer from a number of Universal Forwarders on the network. My indexer is both indexing this data and on-forwarding it to a Heavy Forwarder on my network. The Heavy Forwarder then forwards my log data off to a third party system. This has all been working well.

I am attempting to configure my Heavy Forwarder so that it forwards it's _internal logs back to my indexer but can't get it working.

In order to get the Heavy Forwarder forwarding _internal logs back to my Indexer, I created an app on the Heavy Forwarder /opt/splunk/etc/apps/forward_internal_back2_Indexer. Inside this app I placed the following files:
_____________________________________

default/inputs.conf
[monitor//$SPLUNK_HOME/var/log/splunk/splunkd.log/splunk/splunkd.log]
disabled=0
sourcetype=splunkd
index=_internal

[monitor//$SPLUNK_HOME/var/log/splunk/splunkd.log/splunk/metrics.log]
disabled=0
sourcetype=splunkd
index=_internal
_____________________________________

default/props.conf

[splunkd]
TRANSFORMS-routing=routeBack2Indexer
_____________________________________

default/transforms.conf
[routeBack2Indexer]
REGEX=(.)
DEST_KEY=_TCP_ROUTING
FORMAT=HF_internallogs_to_indexer
_____________________________________

default/outputs.conf
[tcpout:HF_internallogs_to_indexer]
server = <ip_address_of_splunk_indexer>:9997
_____________________________________

Once I had done this I restart splunkd on the Heavy Forwarder, However I can't seem to see _internal logs coming back from my Heavy Forwarder host.

would appreciate some help, figuring out where I've gone wrong 🙂

[mike_k]

@gcusellothanks for that. Will give that a go.

[gcusello]

Hi @mike_k,

ok, let me know.

Ciao.

Giuseppe

P.S.: Karma Points are appreciated 😉

[gcusello]

Hi @mike_k,

you already have an input (in $SPLUNK_HOME/etc/system/default) to take internal logs and Splunk doesn't permits to twice index a log.

So, as described at https://docs.splunk.com/Documentation/Splunk/latest/Forwarding/Routeandfilterdatad#Route_inputs_to_s... try to copy inputs.conf from $SPLUNK_HOME/etc/system/default to $SPLUNK_HOME/etc/system/local and add 

_TCP_ROUTING = HF_internallogs_to_indexer

to the internal logs stanza (and obviously restart Splunk on HF).

Ciao.

Giuseppe