Monitoring Splunk

How to configure heavy forwarder _internal index forwarding?

mike_k
Path Finder

I have a single instance Splunk Enterprise deployment running on Linux. I have a bunch of data feeding into my indexer from a number of Universal Forwarders on the network. My indexer is both indexing this data and on-forwarding it to a Heavy Forwarder on my network. The Heavy Forwarder then forwards my log data off to a third party system. This has all been working well.

I am attempting to configure my Heavy Forwarder so that it forwards it's _internal logs back to my indexer but can't get it working.

In order to get the Heavy Forwarder forwarding _internal logs back to my Indexer, I created an app on the Heavy Forwarder /opt/splunk/etc/apps/forward_internal_back2_Indexer. Inside this app I placed the following files:
_____________________________________

default/inputs.conf
[monitor//$SPLUNK_HOME/var/log/splunk/splunkd.log/splunk/splunkd.log]
disabled=0
sourcetype=splunkd
index=_internal

[monitor//$SPLUNK_HOME/var/log/splunk/splunkd.log/splunk/metrics.log]
disabled=0
sourcetype=splunkd
index=_internal
_____________________________________

default/props.conf

[splunkd]
TRANSFORMS-routing=routeBack2Indexer
_____________________________________

default/transforms.conf
[routeBack2Indexer]
REGEX=(.)
DEST_KEY=_TCP_ROUTING
FORMAT=HF_internallogs_to_indexer
_____________________________________

default/outputs.conf
[tcpout:HF_internallogs_to_indexer]
server = <ip_address_of_splunk_indexer>:9997
_____________________________________

Once I had done this I restart splunkd on the Heavy Forwarder, However I can't seem to see _internal logs coming back from my Heavy Forwarder host.

would appreciate some help, figuring out where I've gone wrong 🙂

Labels (1)
0 Karma

mike_k
Path Finder

@gcusellothanks for that. Will give that a go.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @mike_k,

ok, let me know.

Ciao.

Giuseppe

P.S.: Karma Points are appreciated 😉

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @mike_k,

you already have an input (in $SPLUNK_HOME/etc/system/default) to take internal logs and Splunk doesn't permits to twice index a log.

So, as described at https://docs.splunk.com/Documentation/Splunk/latest/Forwarding/Routeandfilterdatad#Route_inputs_to_s... try to copy inputs.conf from $SPLUNK_HOME/etc/system/default to $SPLUNK_HOME/etc/system/local and add 

_TCP_ROUTING = HF_internallogs_to_indexer

to the internal logs stanza (and obviously restart Splunk on HF).

Ciao.

Giuseppe

0 Karma
Get Updates on the Splunk Community!

Technical Workshop Series: Splunk Data Management and SPL2 | Register here!

Hey, Splunk Community! Ready to take your data management skills to the next level? Join us for a 3-part ...

Spotting Financial Fraud in the Haystack: A Guide to Behavioral Analytics with Splunk

In today's digital financial ecosystem, security teams face an unprecedented challenge. The sheer volume of ...

Solve Problems Faster with New, Smarter AI and Integrations in Splunk Observability

Solve Problems Faster with New, Smarter AI and Integrations in Splunk Observability As businesses scale ...