Getting Data In

Fluentd HEC Output: How to target and utilize parts of a tag to configure my index, sourcetype, and host dynamically?

Engager

I've got a bunch of custom syslog traffic flowing to a fluentd tier I have running in kubernetes. I'm using the rewritetagfilter plugin to set the tag of all the events to their target index. I then use another layer of that plugin to add the host and sourcetype values to the tag.

I'm sending all of that to the same output:

   @type splunk_hec
   index main
   sourcetype ${tag_parts[1]}
   host ${tag_suffix[2]}
   source ${tag}
   hec_host HEC_Host
   hec_port HEC Port
   hec_token HEC Token
   ca_file /fluentd/etc/server.pem

In the configs above I'd like to target different parts of the tag to configure my index, sourcetype, and host dynamically.

The sourcetype and host lines translate those directly to a string, so in Splunk for example I see the host field literally set to "${tag_suffix[2]}"

But the source field I'm setting as a test work and the source field in Splunk contains the whole tag.

How can I target and utilize parts of the tag to configure my settings? Or is there a better way to set these values?
Trying to avoid index time operations on my indexers.

Thanks!

Sources:
I found the prefix, suffix, and parts for tag targeting in record transformer and wasn't sure if they would work
https://docs.fluentd.org/filter/record_transformer

Fluentd to Hec plugin, latest version
https://github.com/splunk/fluent-plugin-splunk-hec

0 Karma