Can a Heavy-Forwarder just raw forward and not parsing?

[verbal_666]

Hi all.

Like the subject, can i tell an HF not to PARSE the events, just do a banal tcp forwarding of the raw data? I can replace an HF with a banal TCP-FORWARDING tool, and it works. But the question is about the HF, since i need to deploy all props/transforms in INDEXER BUT ALSO IN HF, if i do not want to index erroneous events...

I mean, outputs from UF is balanced from real INDEXER and HF (do not question 🙂 )

RIGHT SCENARIO (props/transforms in both Indexer/HF)
UF --> IDX --> parsing --> correct events
UF --> HF --> parsing --> IDX --> correct events

WRONG SCENARIO (props/transforms only in Indexer, not in HF)
UF --> IDX --> parsing --> correct events
UF --> HF (bad event parsing, no timestamp no linebreak etc..) --> IDX --> event already bad parsed --> erroneous events indexed!!!

Thanks.

Bye... 😎

[dokaas_2]

I may get slapped for this answer since Splunk technically says it's deprecated, but you could consider using the LightWeightForwarder.  The LWF disables the parsing & index queues on the HF, but gives one the HEC inputs and Python (for running apps such as eStreamer, dbConnect, etc) as an intermediate relay.   A LWF disables the web interface, but that can be turned back on in a higher precedence web.conf.

I use a LWF in the DMZ which gives me all the above, but also provides a deployment server for assets in the DMZ.

[gseyerl]

Hi Kundeng,
how exactly did you manage to setup you IHF in order for the downstream indexers to parse the data. How did you set the route keyword?

Best
G

[verbal_666]

So... teorically, i should apply

queue = indexQueue

in inputs.conf of the HF... to avoid parsing (default), and raw forward the event... this should do the trick...

Thanks a lot 👍

[venkatasri]

@verbal_666 yes if you are using HF as forwarder without UF. I guess it should be on the very first forwarding layer. Please test and let me know interested to see how this works, haven't implemented though.

[venkatasri · Accepted solution]

Hi @verbal_666 

As docs says you might restrict queues in input layer, meaning on UF if you are using it to forward data.

On UF inputs.conf to the monitor you wanted to avoid parsing, set queue = indexQueue. I would advise to test it before and do not set at global unless you really wanted. Refer specs - inputs.conf - Splunk Documentation

----

An upvote would be appreciated and Accept solution if it helps!

queue = [parsingQueue|indexQueue]
* Sets the queue where the input processor should deposit the events it reads.
* Set to "parsingQueue" to apply the props.conf file and other parsing rules to
  your data. For more information about the props.conf file and rules
  timestamps and linebreaks, see the props.conf file and the
  online documentation at http://docs.splunk.com/Documentation.
* Set to "indexQueue" to send your data directly into the index.
* Default: parsingQueue

[kundeng]

In the documentation and answer, it talks about UF; the original question seems to be about HF. 

For Intermediate HF (IHF),  this is NOT the way for forcing the downstream indexers to be the place of parsing data.  I tried it with Splunk cloud deployment.   

Downstream indexers (aka cloud indexers) will NOT parse the data (index time timestamp fixing, transforms etc) without other techniques (setting route keyword).