topic Re: Adding Index-time field on intermediate forwarder in Splunk Search

Adding Index-time field on intermediate forwarder

greich — Tue, 29 Sep 2020 06:41:55 GMT

I need to trace the data from the originating forwarder through intermediate forwarders or directly onto indexers. I am trying to add a field when the data goes through the intermediate forwarder.

Configuration

Data collection by Heavy Forwarders

inputs.conf
[default]
host = shorthostname (of the HF)

sending all to Intermediate Forwarders

inputs.conf
[default]
host = shorthostname (IF)
router = shorthostname (IF)

props.conf
[default]
TRANSFORMS-router = addrouter

transforms.conf
[addrouter]
SOURCE_KEY = router
REGEX = (.*)
FORMAT = router::$1
WRITE_META = true

[accepted_keys]
DCIF_NAME = router

And finally on the search head
fields.conf
[router]
INDEXED = true

It seems that:
1- everything that is sourced directly on the IF (syslog and splunk logs) has the field "router"
2- everything incoming on TCP from HF does not have the field "router"

What I am missing to mark incoming cooked data?

Re: Adding Index-time field on intermediate forwarder

greich — Wed, 15 Jul 2015 14:55:17 GMT

I must mention that a start message:
Checking conf files for problems...
Invalid key in stanza [default] in /opt/splunk/etc/system/local/inputs.conf, line 3: router (value: blahhost)

Re: Adding Index-time field on intermediate forwarder

jacobwilkins — Tue, 29 Sep 2020 06:41:41 GMT

This is expected behavior. You are going to have to re-parse your cooked data:

See here:
http://answers.splunk.com/answers/97918/reparsing-cooked-data-coming-from-a-heavy-forwarder-possible.html

On your IF (which probably needs to be a HWF), do this in inputs.conf:

[splunktcp]
route=has_key:_utf8:parsingQueue;has_key:_linebreaker:parsingQueue;absent_key:_utf8:parsingQueue;absent_key:_linebreaker:parsingQueue

Re: Adding Index-time field on intermediate forwarder

greich — Wed, 15 Jul 2015 15:53:52 GMT

Yes IF are HF. On the first instance, your solution seems to work perfectly. Will let soak, and confirm later. Thank you.

Re: Adding Index-time field on intermediate forwarder

greich — Thu, 16 Jul 2015 08:15:23 GMT

that did it, with no noticeable impact on CPU on the IF (was expecting some). Thanks again

Re: Adding Index-time field on intermediate forwarder

greich — Wed, 17 Feb 2016 09:55:24 GMT

adding the key to the .conf.spec should resolve this