Monitoring Splunk

How to configure heavy forwarder _internal index forwarding?

mike_k
Path Finder

I have a single instance Splunk Enterprise deployment running on Linux. I have a bunch of data feeding into my indexer from a number of Universal Forwarders on the network. My indexer is both indexing this data and on-forwarding it to a Heavy Forwarder on my network. The Heavy Forwarder then forwards my log data off to a third party system. This has all been working well.

I am attempting to configure my Heavy Forwarder so that it forwards it's _internal logs back to my indexer but can't get it working.

In order to get the Heavy Forwarder forwarding _internal logs back to my Indexer, I created an app on the Heavy Forwarder /opt/splunk/etc/apps/forward_internal_back2_Indexer. Inside this app I placed the following files:
_____________________________________

default/inputs.conf
[monitor//$SPLUNK_HOME/var/log/splunk/splunkd.log/splunk/splunkd.log]
disabled=0
sourcetype=splunkd
index=_internal

[monitor//$SPLUNK_HOME/var/log/splunk/splunkd.log/splunk/metrics.log]
disabled=0
sourcetype=splunkd
index=_internal
_____________________________________

default/props.conf

[splunkd]
TRANSFORMS-routing=routeBack2Indexer
_____________________________________

default/transforms.conf
[routeBack2Indexer]
REGEX=(.)
DEST_KEY=_TCP_ROUTING
FORMAT=HF_internallogs_to_indexer
_____________________________________

default/outputs.conf
[tcpout:HF_internallogs_to_indexer]
server = <ip_address_of_splunk_indexer>:9997
_____________________________________

Once I had done this I restart splunkd on the Heavy Forwarder, However I can't seem to see _internal logs coming back from my Heavy Forwarder host.

would appreciate some help, figuring out where I've gone wrong 🙂

Labels (1)
0 Karma

mike_k
Path Finder

@gcusellothanks for that. Will give that a go.

0 Karma

gcusello
Esteemed Legend

Hi @mike_k,

ok, let me know.

Ciao.

Giuseppe

P.S.: Karma Points are appreciated 😉

0 Karma

gcusello
Esteemed Legend

Hi @mike_k,

you already have an input (in $SPLUNK_HOME/etc/system/default) to take internal logs and Splunk doesn't permits to twice index a log.

So, as described at https://docs.splunk.com/Documentation/Splunk/latest/Forwarding/Routeandfilterdatad#Route_inputs_to_s... try to copy inputs.conf from $SPLUNK_HOME/etc/system/default to $SPLUNK_HOME/etc/system/local and add 

_TCP_ROUTING = HF_internallogs_to_indexer

to the internal logs stanza (and obviously restart Splunk on HF).

Ciao.

Giuseppe

0 Karma
Get Updates on the Splunk Community!

Splunk Community Platform Survey

Hey Splunk Community, Starting today, the community platform may prompt you to participate in a survey. The ...

Observability Highlights | November 2022 Newsletter

 November 2022Observability CloudEnd Of Support Extension for SignalFx Smart AgentSplunk is extending the End ...

Avoid Certificate Expiry Issues in Splunk Enterprise with Certificate Assist

This blog post is part 2 of 4 of a series on Splunk Assist. Click the links below to see the other ...