Deployment Architecture

Sending cooked but unparsed data from Heavy Forwarder

nicolas_perreau
Explorer

Hi Splunkers,

We are ingesting data using the Splunk TA for AWS, which is installed on a heavy forwarder. While this works great within Splunk, we'd like to forward data from the indexer cluster to a 3rd party system using a props.conf sourcetype match and a transforms.conf regex to route the specific events. We've done that numerous time and it works well for other sources (coming from Universal Forwarders).

Here's our ingestion pipeline for AWS events :

AWS S3 <- Splunk TA AWS (on HF) -> IXC -> 3rd party system

Unfortunately, we can't find a way at this point to route events based on the sourcetype at the indexing layer. Our understanding is that the HF will cook and parse the events and the indexer will skip to the indexing queue directly.

The question is : is there any way to get the data from the HF to be sent unparsed but cooked, exactly the same way the UF does so that the indexing layer will be able to parse the events through all the pipelines?

Thanks!

0 Karma

gjanders
SplunkTrust
SplunkTrust

You could try sendCookedData = false in outputs.conf but then you would need to receive the data via a non-Splunk input (tcp or similar)!

But why wouldn't you just do the transforms to send the data to the 3rd party on the heavy forwarder itself?

0 Karma

nicolas_perreau
Explorer

HI gjanders.

I'd like to mimic the same behavior that the UF has from the HF. The only reason I need the UF is to get the AWS TA working, otherwise I would rather like to use a UF.

Thanks for your suggestions!

0 Karma

somesoni2
Revered Legend

Though it's possible to have your indexers reparse every data it receives (even though it's already parsed through HF already), but this can only be done at global level and not for specific sourcetypes/sources/hosts etc. Why don't you do that forwarding to 3rd party system at HF level itself?

If you still want to know about enable reparsing, look at this
https://answers.splunk.com/answers/97918/reparsing-cooked-data-coming-from-a-heavy-forwarder-possibl...
https://docs.splunk.com/Documentation/Splunk/latest/Admin/Inputsconf#Data_distribution:

0 Karma

nicolas_perreau
Explorer

Hi somesoni2,

Thanks for your answer. Sending from the UF could be done, but we'd rather like to keep the same ingestion and forwarding pipeline that is already defined (from the indexing layer).

I guess that I'll have to fall back on that solution if there is no way to disable the parsing of event from the HF directly.

Thanks again!

0 Karma
Get Updates on the Splunk Community!

Happy CX Day to our Community Superheroes!

Happy 10th Birthday CX Day!What is CX Day? It’s a global celebration recognizing innovation and success in the ...

Check out This Month’s Brand new Splunk Lantern Articles

Splunk Lantern is a customer success center providing advice from Splunk experts on valuable data insights, ...

Routing Data to Different Splunk Indexes in the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...