topic concatenate a field to my source and regex the result in Splunk ITSI

concatenate a field to my source and regex the result

ictrees28 — Tue, 22 Dec 2020 16:11:22 GMT

Splunk Noob.
I have a custom http sourcetype with multiple data sources. For one of these sources (aws:firehose), I need to concatenate a field value (ecs_task_definition) to the source value, then do a regex or an eval at some point to remove the trailing colon and numbers, preferably all at index time. I've been advised the field=ecs_task_definition will contain a few hundred dynamic values that will change from time to time, so I can't assign these statically.

My example:

sourcetype=httpevent

source=aws:billing
source=aws:s3
source=aws:inspector
source=aws:firehose

ecs_task_definition=arc-permission-service-worker:100
ecs_task_definition=arc-enrollment-service:182
ecs_task_definition=arc-reporting-service:234
ecs_task_definition=arc-tenant-service:332

I would like the final result to look like:

source=aws:firehose:arc-tenant-service
source=aws:firehose:arc-reporting-service

I have been trying to do this in props and transforms without success. I think I'm having both syntax problems added to a general lack of understanding of what I can and can't do at index time verses search time. Any help would be much appreciated. Thanks

Re: concatenate a field to my source and regex the result

scelikok — Tue, 22 Dec 2020 17:02:03 GMT

@ictrees28, if you are using version 7.2 or later, you can create or update any field with INGEST_EVAL easily. Since you will able to use the same syntax as EVAL, you can test your EVAL on search then apply to transform .

Sample;

props.conf [httpevent] TRANSFORMS-update_source = update_source transforms.conf [update_source] INGEST_EVAL = source:=source.":".mvindex(split(ecs_task_definition,":"),0)

Re: concatenate a field to my source and regex the result

ictrees28 — Tue, 22 Dec 2020 19:08:14 GMT

Thank you so much, that makes perfect sense. That said, it's still not extracting my ecs_task_definition field.

The internal index error message reads "Cannot parse INGEST_EVAL statement "source: = source.":".mvindex(split(ecs_task_definition,":"),0)" into component parts for update_source

Not sure what I'm doing wrong.