We are ingesting data using the Splunk TA for AWS, which is installed on a heavy forwarder. While this works great within Splunk, we'd like to forward data from the indexer cluster to a 3rd party system using a props.conf sourcetype match and a transforms.conf regex to route the specific events. We've done that numerous time and it works well for other sources (coming from Universal Forwarders).
Here's our ingestion pipeline for AWS events :
AWS S3 <- Splunk TA AWS (on HF) -> IXC -> 3rd party system
Unfortunately, we can't find a way at this point to route events based on the sourcetype at the indexing layer. Our understanding is that the HF will cook and parse the events and the indexer will skip to the indexing queue directly.
The question is : is there any way to get the data from the HF to be sent unparsed but cooked, exactly the same way the UF does so that the indexing layer will be able to parse the events through all the pipelines?
Though it's possible to have your indexers reparse every data it receives (even though it's already parsed through HF already), but this can only be done at global level and not for specific sourcetypes/sources/hosts etc. Why don't you do that forwarding to 3rd party system at HF level itself?
If you still want to know about enable reparsing, look at this
Thanks for your answer. Sending from the UF could be done, but we'd rather like to keep the same ingestion and forwarding pipeline that is already defined (from the indexing layer).
I guess that I'll have to fall back on that solution if there is no way to disable the parsing of event from the HF directly.
You could try sendCookedData = false in outputs.conf but then you would need to receive the data via a non-Splunk input (tcp or similar)!
But why wouldn't you just do the transforms to send the data to the 3rd party on the heavy forwarder itself?
I'd like to mimic the same behavior that the UF has from the HF. The only reason I need the UF is to get the AWS TA working, otherwise I would rather like to use a UF.
Thanks for your suggestions!