Splunk ITSI

concatenate a field to my source and regex the result

ictrees28
Loves-to-Learn Everything

Splunk Noob.
I have a custom http sourcetype with multiple data sources. For one of these sources (aws:firehose), I need to concatenate a field value (ecs_task_definition) to the source value, then do a regex or an eval at some point to remove the trailing colon and numbers, preferably all at index time. I've been advised the field=ecs_task_definition will contain a few hundred dynamic values that will change from time to time, so I can't assign these statically.

My example:

sourcetype=httpevent

source=aws:billing
source=aws:s3
source=aws:inspector
source=aws:firehose

ecs_task_definition=arc-permission-service-worker:100
ecs_task_definition=arc-enrollment-service:182
ecs_task_definition=arc-reporting-service:234
ecs_task_definition=arc-tenant-service:332

I would like the final result to look like:

source=aws:firehose:arc-tenant-service
source=aws:firehose:arc-reporting-service


I have been trying to do this in props and transforms without success. I think I'm having both syntax problems added to a general lack of understanding of what I can and can't do at index time verses search time. Any help would be much appreciated. Thanks

Tags (2)
0 Karma

scelikok
SplunkTrust
SplunkTrust

@ictrees28, if you are using version 7.2 or later, you can create or update any field with INGEST_EVAL easily. Since you will able to use the same syntax as EVAL, you can test your EVAL on search then apply to transform .

Sample;

props.conf
[httpevent]
TRANSFORMS-update_source = update_source

transforms.conf
[update_source]
INGEST_EVAL = source:=source.":".mvindex(split(ecs_task_definition,":"),0)

 

If this reply helps you an upvote and "Accept as Solution" is appreciated.
0 Karma

ictrees28
Loves-to-Learn Everything

Thank you so much, that makes perfect sense.   That said, it's still not extracting my ecs_task_definition field. 

The internal index error message reads "Cannot parse INGEST_EVAL statement "source: = source.":".mvindex(split(ecs_task_definition,":"),0)" into component parts for update_source

Not sure what I'm doing wrong. 

0 Karma
Get Updates on the Splunk Community!

Splunk Observability for AI

Don’t miss out on an exciting Tech Talk on Splunk Observability for AI!Discover how Splunk’s agentic AI ...

Splunk Enterprise Security 8.x: The Essential Upgrade for Threat Detection, ...

Watch On Demand the Tech Talk on November 6 at 11AM PT, and empower your SOC to reach new heights! Duration: ...

Splunk Observability as Code: From Zero to Dashboard

For the details on what Self-Service Observability and Observability as Code is, we have some awesome content ...