Getting Data In

Can a Heavy-Forwarder just raw forward and not parsing?

verbal_666
Builder

Hi all.

Like the subject, can i tell an HF not to PARSE the events, just do a banal tcp forwarding of the raw data? I can replace an HF with a banal TCP-FORWARDING tool, and it works. But the question is about the HF, since i need to deploy all props/transforms in INDEXER BUT ALSO IN HF, if i do not want to index erroneous events...

 

I mean, outputs from UF is balanced from real INDEXER and HF (do not question 🙂 )

RIGHT SCENARIO (props/transforms in both Indexer/HF)
UF --> IDX --> parsing --> correct events
UF --> HF --> parsing --> IDX --> correct events

WRONG SCENARIO (props/transforms only in Indexer, not in HF)
UF --> IDX --> parsing --> correct events
UF --> HF (bad event parsing, no timestamp no linebreak etc..) --> IDX --> event already bad parsed --> erroneous events indexed!!!

 

Thanks.

Bye... 😎

Labels (1)
0 Karma
1 Solution

venkatasri
SplunkTrust
SplunkTrust

Hi @verbal_666 

As docs says you might restrict queues in input layer, meaning on UF if you are using it to forward data.

On UF inputs.conf to the monitor you wanted to avoid parsing, set queue = indexQueue. I would advise to test it before and do not set at global unless you really wanted. Refer specs - inputs.conf - Splunk Documentation

----

An upvote would be appreciated and Accept solution if it helps!

 

queue = [parsingQueue|indexQueue]
* Sets the queue where the input processor should deposit the events it reads.
* Set to "parsingQueue" to apply the props.conf file and other parsing rules to
  your data. For more information about the props.conf file and rules
  timestamps and linebreaks, see the props.conf file and the
  online documentation at http://docs.splunk.com/Documentation.
* Set to "indexQueue" to send your data directly into the index.
* Default: parsingQueue

 

 

 

View solution in original post

dokaas_2
Path Finder

I may get slapped for this answer since Splunk technically says it's deprecated, but you could consider using the LightWeightForwarder.  The LWF disables the parsing & index queues on the HF, but gives one the HEC inputs and Python (for running apps such as eStreamer, dbConnect, etc) as an intermediate relay.   A LWF disables the web interface, but that can be turned back on in a higher precedence web.conf.

I use a LWF in the DMZ which gives me all the above, but also provides a deployment server for assets in the DMZ.

0 Karma

gseyerl
Loves-to-Learn Lots

Hi Kundeng,
how exactly did you manage to setup you IHF in order for the downstream indexers to parse the data. How did you set the route keyword?

Best
G

0 Karma

verbal_666
Builder

So... teorically, i should apply

queue = indexQueue

in inputs.conf of the HF... to avoid parsing (default), and raw forward the event... this should do the trick...

Thanks a lot 👍

0 Karma

venkatasri
SplunkTrust
SplunkTrust

@verbal_666 yes if you are using HF as forwarder without UF. I guess it should be on the very first forwarding layer. Please test and let me know interested to see how this works, haven't implemented though.

0 Karma

venkatasri
SplunkTrust
SplunkTrust

Hi @verbal_666 

As docs says you might restrict queues in input layer, meaning on UF if you are using it to forward data.

On UF inputs.conf to the monitor you wanted to avoid parsing, set queue = indexQueue. I would advise to test it before and do not set at global unless you really wanted. Refer specs - inputs.conf - Splunk Documentation

----

An upvote would be appreciated and Accept solution if it helps!

 

queue = [parsingQueue|indexQueue]
* Sets the queue where the input processor should deposit the events it reads.
* Set to "parsingQueue" to apply the props.conf file and other parsing rules to
  your data. For more information about the props.conf file and rules
  timestamps and linebreaks, see the props.conf file and the
  online documentation at http://docs.splunk.com/Documentation.
* Set to "indexQueue" to send your data directly into the index.
* Default: parsingQueue

 

 

 

kundeng
Path Finder

In the documentation and answer, it talks about UF; the original question seems to be about HF. 

For Intermediate HF (IHF),  this is NOT the way for forcing the downstream indexers to be the place of parsing data.  I tried it with Splunk cloud deployment.   

Downstream indexers (aka cloud indexers) will NOT parse the data (index time timestamp fixing, transforms etc) without other techniques (setting route keyword).  

 

 

Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...