Splunk Search

Adding Index-time field on intermediate forwarder

Communicator

I need to trace the data from the originating forwarder through intermediate forwarders or directly onto indexers. I am trying to add a field when the data goes through the intermediate forwarder.

Configuration

Data collection by Heavy Forwarders

inputs.conf
[default]
host = shorthostname (of the HF)

sending all to Intermediate Forwarders

inputs.conf
[default]
host = shorthostname (IF)
router = shorthostname (IF)

props.conf
[default]
TRANSFORMS-router = addrouter

transforms.conf
[addrouter]
SOURCEKEY = router
REGEX = (.*)
FORMAT = router::$1
WRITE
META = true

[acceptedkeys]
DCIF
NAME = router

And finally on the search head
fields.conf
[router]
INDEXED = true

It seems that:
1- everything that is sourced directly on the IF (syslog and splunk logs) has the field "router"
2- everything incoming on TCP from HF does not have the field "router"

What I am missing to mark incoming cooked data?

0 Karma
1 Solution

Communicator

This is expected behavior. You are going to have to re-parse your cooked data:

See here:
http://answers.splunk.com/answers/97918/reparsing-cooked-data-coming-from-a-heavy-forwarder-possible...

On your IF (which probably needs to be a HWF), do this in inputs.conf:

[splunktcp]
route=haskey:utf8:parsingQueue;haskey:linebreaker:parsingQueue;absentkey:utf8:parsingQueue;absentkey:linebreaker:parsingQueue

View solution in original post

0 Karma

Communicator

This is expected behavior. You are going to have to re-parse your cooked data:

See here:
http://answers.splunk.com/answers/97918/reparsing-cooked-data-coming-from-a-heavy-forwarder-possible...

On your IF (which probably needs to be a HWF), do this in inputs.conf:

[splunktcp]
route=haskey:utf8:parsingQueue;haskey:linebreaker:parsingQueue;absentkey:utf8:parsingQueue;absentkey:linebreaker:parsingQueue

View solution in original post

0 Karma

Communicator

Yes IF are HF. On the first instance, your solution seems to work perfectly. Will let soak, and confirm later. Thank you.

0 Karma

Communicator

that did it, with no noticeable impact on CPU on the IF (was expecting some). Thanks again

0 Karma

Communicator

I must mention that a start message:
Checking conf files for problems...
Invalid key in stanza [default] in /opt/splunk/etc/system/local/inputs.conf, line 3: router (value: blahhost)

0 Karma

Communicator

adding the key to the .conf.spec should resolve this

0 Karma