Splunk Install in Forwarder Mode and Props

[ifeldshteyn]

Hi,

I want to preface I understand that props isn't fully processed if you install it on the universal forwarder. My question is about the difference between the install of a Splunk Universal Forwarder vs Splunk converted to a Forwarder license.

My setup is a fresh install of an ancient 6.5.2 SplunkForwarder and 6.5.2 Splunk in Forwarder Mode on two machines and mapped like this.

Server1: Universal Forwarder --> Indexer
Server2: Splunk (updated to run as a Forwarder) --> Indexer.

On the indexer I have a props change (a trivial SEDCMD-test = s/a/o/g )

If I install the same serverclass on both servers that reads a /tmp/test.log and where I write some lines with letters a in them the Server 1's messages end up changed from a to o while Server 2's do not, they staay as a. I've tested it with multiple server installs (albeit on an old version 6.5.2).

It seems to me that Splunk in forwarder, unlike a dedicated Splunk Universal Forwarder, applies some kind of tag that prevents downstream props/transforms changes to occur. A raw message coming from Universal Forwarder is then processed by the indexer's props/transforms while a message coming from Splunk in Forwarder mode does not.

Note: I checked for any silliness. Both Servers send to the same indexer, have identical serverclass, sourcetype, inputs and outputs. And the props on the indexer only applies to a sourcetype not any specific host (btool matches on source servers)

My questions are listed below.

1. Can you confirm that there is some kind of cooked tag on events coming from Splunk (in forwarder mode) that tells downstream systems not to apply props/transforms and just write immediately to an index?

2. Is there anything I can do on the server with Splunk in forwarder mode to behave exactly like a UniversalForwarder? Perhaps a etc conf change, or do I need to just uninstall it and setup Splunk UniversalForwarder from scratch (confirmed this works).

3. How can I debug props/transforms issues between two servers. Turning on DEBUG mode didn't say anything useful in splunkd.log like "applying props [foo] on event 'Hello World'"

Thanks!

[richgalloway · Accepted solution]

Splunk (in forwarder mode) is called a Heavy Forwarder or HF. HFs do the work of an indexer, but do not store data. The indexer sees that the events have already been processed and just write the data to disk.

There is no way to tell an HF to behave like a Universal Forwarder (UF). The HF will always process events and the UF will do very little processing.

There is nothing that tells us how events are processed. We can only look at the input and the output and guess at what happened in between.

In your environment, props.conf files should be installed on all instances - UF, HF, indexer, and search head.

[ifeldshteyn]

Thank you Rich. I guess this means we cannot layer Heavy Forwarders vertically if we have a complicated props/transforms. This can only be done by installing more Heavy Forwarders horizontally.

[richgalloway]

Props and transforms are processed in one place, HF or indexer which first touches the data (with exceptions we won't discuss here), regardless of complexity. "Complicated" props cannot be split among instances.