Getting Data In

Can an Heavy-Forwarder just raw forward and not parsing?

verbal_666
Communicator

Hi all.

Like the subject, can i tell an HF not to PARSE the events, just do a banal tcp forwarding of the raw data? I can replace an HF with a banal TCP-FORWARDING tool, and it works. But the question is about the HF, since i need to deploy all props/transforms in INDEXER BUT ALSO IN HF, if i do not want to index erroneous events...

 

I mean, outputs from UF is balanced from real INDEXER and HF (do not question 🙂 )

RIGHT SCENARIO (props/transforms in both Indexer/HF)
UF --> IDX --> parsing --> correct events
UF --> HF --> parsing --> IDX --> correct events

WRONG SCENARIO (props/transforms only in Indexer, not in HF)
UF --> IDX --> parsing --> correct events
UF --> HF (bad event parsing, no timestamp no linebreak etc..) --> IDX --> event already bad parsed --> erroneous events indexed!!!

 

Thanks.

Bye... 😎

Labels (1)
0 Karma
1 Solution

venkatasri
SplunkTrust
SplunkTrust

Hi @verbal_666 

As docs says you might restrict queues in input layer, meaning on UF if you are using it to forward data.

On UF inputs.conf to the monitor you wanted to avoid parsing, set queue = indexQueue. I would advise to test it before and do not set at global unless you really wanted. Refer specs - inputs.conf - Splunk Documentation

----

An upvote would be appreciated and Accept solution if it helps!

 

queue = [parsingQueue|indexQueue]
* Sets the queue where the input processor should deposit the events it reads.
* Set to "parsingQueue" to apply the props.conf file and other parsing rules to
  your data. For more information about the props.conf file and rules
  timestamps and linebreaks, see the props.conf file and the
  online documentation at http://docs.splunk.com/Documentation.
* Set to "indexQueue" to send your data directly into the index.
* Default: parsingQueue

 

 

 

View solution in original post

verbal_666
Communicator

So... teorically, i should apply

queue = indexQueue

in inputs.conf of the HF... to avoid parsing (default), and raw forward the event... this should do the trick...

Thanks a lot 👍

0 Karma

venkatasri
SplunkTrust
SplunkTrust

@verbal_666 yes if you are using HF as forwarder without UF. I guess it should be on the very first forwarding layer. Please test and let me know interested to see how this works, haven't implemented though.

0 Karma

venkatasri
SplunkTrust
SplunkTrust

Hi @verbal_666 

As docs says you might restrict queues in input layer, meaning on UF if you are using it to forward data.

On UF inputs.conf to the monitor you wanted to avoid parsing, set queue = indexQueue. I would advise to test it before and do not set at global unless you really wanted. Refer specs - inputs.conf - Splunk Documentation

----

An upvote would be appreciated and Accept solution if it helps!

 

queue = [parsingQueue|indexQueue]
* Sets the queue where the input processor should deposit the events it reads.
* Set to "parsingQueue" to apply the props.conf file and other parsing rules to
  your data. For more information about the props.conf file and rules
  timestamps and linebreaks, see the props.conf file and the
  online documentation at http://docs.splunk.com/Documentation.
* Set to "indexQueue" to send your data directly into the index.
* Default: parsingQueue

 

 

 

View solution in original post

Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!