Splunk ITSI

concatenate a field to my source and regex the result

ictrees28
Loves-to-Learn Everything

Splunk Noob.
I have a custom http sourcetype with multiple data sources. For one of these sources (aws:firehose), I need to concatenate a field value (ecs_task_definition) to the source value, then do a regex or an eval at some point to remove the trailing colon and numbers, preferably all at index time. I've been advised the field=ecs_task_definition will contain a few hundred dynamic values that will change from time to time, so I can't assign these statically.

My example:

sourcetype=httpevent

source=aws:billing
source=aws:s3
source=aws:inspector
source=aws:firehose

ecs_task_definition=arc-permission-service-worker:100
ecs_task_definition=arc-enrollment-service:182
ecs_task_definition=arc-reporting-service:234
ecs_task_definition=arc-tenant-service:332

I would like the final result to look like:

source=aws:firehose:arc-tenant-service
source=aws:firehose:arc-reporting-service


I have been trying to do this in props and transforms without success. I think I'm having both syntax problems added to a general lack of understanding of what I can and can't do at index time verses search time. Any help would be much appreciated. Thanks

Tags (2)
0 Karma

scelikok
SplunkTrust
SplunkTrust

@ictrees28, if you are using version 7.2 or later, you can create or update any field with INGEST_EVAL easily. Since you will able to use the same syntax as EVAL, you can test your EVAL on search then apply to transform .

Sample;

props.conf
[httpevent]
TRANSFORMS-update_source = update_source

transforms.conf
[update_source]
INGEST_EVAL = source:=source.":".mvindex(split(ecs_task_definition,":"),0)

 

If this reply helps you an upvote and "Accept as Solution" is appreciated.
0 Karma

ictrees28
Loves-to-Learn Everything

Thank you so much, that makes perfect sense.   That said, it's still not extracting my ecs_task_definition field. 

The internal index error message reads "Cannot parse INGEST_EVAL statement "source: = source.":".mvindex(split(ecs_task_definition,":"),0)" into component parts for update_source

Not sure what I'm doing wrong. 

0 Karma
Get Updates on the Splunk Community!

Developer Spotlight with Brett Adams

In our third Spotlight feature, we're excited to shine a light on Brett—a Splunk consultant, innovative ...

Index This | What can you do to make 55,555 equal 500?

April 2025 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with this ...

Say goodbye to manually analyzing phishing and malware threats with Splunk Attack ...

In today’s evolving threat landscape, we understand you’re constantly bombarded with phishing and malware ...