topic Does the Universal Forwarder support the Splunk header in Getting Data In

Does the Universal Forwarder support the Splunk header

Marinus — Mon, 03 Sep 2012 09:51:27 GMT

I recently moved to the universal forwarder (4.3.3) where I collect files using the batch input. It's a long story but I have to use the batch input. I use the SPLUNK header to set the host, source and source type. The receiving indexer performs the necessary transformations. I've noticed since I've moved to the new forwarder that the header is no longer being honoured. I changed the HEADER_MODE to always in the default etc/system/local/props.conf, however events arrive without the necessary host, source and sourcetypes.

It appears that the batch handling changed in 4.2, and is not handled as part of the indexing pipeline.

Re: Does the Universal Forwarder support the Splunk header

dart — Tue, 04 Sep 2012 15:25:08 GMT

What's the sourcetype of your data?

Do you have any transforms of the data? What kind of stanza specification are you using on the indexer for these? [my_sourcetype] or [source::/path/to/file] or [host::host1]?

What are you setting on the forwarder inputs?

Re: Does the Universal Forwarder support the Splunk header

Marinus — Mon, 28 Sep 2020 12:22:42 GMT

Thanks for the response Dart. The indexer uses a batch input to collect data.

[batch:///data]
move_policy=sinkhole
crcSalt=

The host, source and sourcetype are set by the splunk header i.e.
SPLUNK host=acme source=xyz sourcetype=abc

The indexer received the events from the forwarder and has props configured to deal with the source types, which in fact rewrite the source and host keys i.e.

[abc]
TRANSFORMS-fix=fix_a, fix_b

When I look at the events on the indexer, I can see that raw events including the SPLUNK header, with no keys set.

Re: Does the Universal Forwarder support the Splunk header

dart — Tue, 04 Sep 2012 19:36:46 GMT

Hi Marinus,
I'm not sure if using the ***SPLUNK*** style is supported.
I'd suggest either using the Splunk Forwarder instead of the universal forwarder, or you could set a sourcetype in your batch input, and reference that sourcetype in the TRANSFORMS, which could fix host, source and sourcetype, and also use a SEDCMD to remove the header.

I'd say the better solution is to use a full forwarder, if that works for ***SPLUNK*** style.

dart

Re: Does the Universal Forwarder support the Splunk header

Marinus — Wed, 05 Sep 2012 12:51:20 GMT

Hi Dart

I did a couple of tests and it doesn't appear that HEADER_MODE config affects they way it processes events 😞

Re: Does the Universal Forwarder support the Splunk header

Marinus — Mon, 22 Oct 2012 08:36:33 GMT

It appears that the batch handling changed in 4.2, and is not handled as part of the indexing pipeline. Reverted back to an old forwarder.