We are ingesting data using the Splunk TA for AWS, which is installed on a heavy forwarder. While this works great within Splunk, we'd like to forward data from the indexer cluster to a 3rd party system using a props.conf sourcetype match and a transforms.conf regex to route the specific events. We've done that numerous time and it works well for other sources (coming from Universal Forwarders).
Here's our ingestion pipeline for AWS events :
AWS S3 <- Splunk TA AWS (on HF) -> IXC -> 3rd party system
Unfortunately, we can't find a way at this point to route events based on the sourcetype at the indexing layer. Our understanding is that the HF will cook and parse the events and the indexer will skip to the indexing queue directly.
The question is : is there any way to get the data from the HF to be sent unparsed but cooked, exactly the same way the UF does so that the indexing layer will be able to parse the events through all the pipelines?
Though it's possible to have your indexers reparse every data it receives (even though it's already parsed through HF already), but this can only be done at global level and not for specific sourcetypes/sources/hosts etc. Why don't you do that forwarding to 3rd party system at HF level itself?