Splunk IT Service Intelligence

concatenate a field to my source and regex the result

ictrees28
Loves-to-Learn Everything

Splunk Noob.
I have a custom http sourcetype with multiple data sources. For one of these sources (aws:firehose), I need to concatenate a field value (ecs_task_definition) to the source value, then do a regex or an eval at some point to remove the trailing colon and numbers, preferably all at index time. I've been advised the field=ecs_task_definition will contain a few hundred dynamic values that will change from time to time, so I can't assign these statically.

My example:

sourcetype=httpevent

source=aws:billing
source=aws:s3
source=aws:inspector
source=aws:firehose

ecs_task_definition=arc-permission-service-worker:100
ecs_task_definition=arc-enrollment-service:182
ecs_task_definition=arc-reporting-service:234
ecs_task_definition=arc-tenant-service:332

I would like the final result to look like:

source=aws:firehose:arc-tenant-service
source=aws:firehose:arc-reporting-service


I have been trying to do this in props and transforms without success. I think I'm having both syntax problems added to a general lack of understanding of what I can and can't do at index time verses search time. Any help would be much appreciated. Thanks

Labels (1)
Tags (2)
0 Karma

scelikok
Influencer

@ictrees28, if you are using version 7.2 or later, you can create or update any field with INGEST_EVAL easily. Since you will able to use the same syntax as EVAL, you can test your EVAL on search then apply to transform .

Sample;

props.conf
[httpevent]
TRANSFORMS-update_source = update_source

transforms.conf
[update_source]
INGEST_EVAL = source:=source.":".mvindex(split(ecs_task_definition,":"),0)

 

If this reply helps you an upvote is appreciated.
0 Karma

ictrees28
Loves-to-Learn Everything

Thank you so much, that makes perfect sense.   That said, it's still not extracting my ecs_task_definition field. 

The internal index error message reads "Cannot parse INGEST_EVAL statement "source: = source.":".mvindex(split(ecs_task_definition,":"),0)" into component parts for update_source

Not sure what I'm doing wrong. 

0 Karma