- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
concatenate a field to my source and regex the result
Splunk Noob.
I have a custom http sourcetype with multiple data sources. For one of these sources (aws:firehose), I need to concatenate a field value (ecs_task_definition) to the source value, then do a regex or an eval at some point to remove the trailing colon and numbers, preferably all at index time. I've been advised the field=ecs_task_definition will contain a few hundred dynamic values that will change from time to time, so I can't assign these statically.
My example:
sourcetype=httpevent
source=aws:billing
source=aws:s3
source=aws:inspector
source=aws:firehose
ecs_task_definition=arc-permission-service-worker:100
ecs_task_definition=arc-enrollment-service:182
ecs_task_definition=arc-reporting-service:234
ecs_task_definition=arc-tenant-service:332
I would like the final result to look like:
source=aws:firehose:arc-tenant-service
source=aws:firehose:arc-reporting-service
I have been trying to do this in props and transforms without success. I think I'm having both syntax problems added to a general lack of understanding of what I can and can't do at index time verses search time. Any help would be much appreciated. Thanks
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content

@ictrees28, if you are using version 7.2 or later, you can create or update any field with INGEST_EVAL easily. Since you will able to use the same syntax as EVAL, you can test your EVAL on search then apply to transform .
Sample;
props.conf
[httpevent]
TRANSFORMS-update_source = update_source
transforms.conf
[update_source]
INGEST_EVAL = source:=source.":".mvindex(split(ecs_task_definition,":"),0)
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Thank you so much, that makes perfect sense. That said, it's still not extracting my ecs_task_definition field.
The internal index error message reads "Cannot parse INGEST_EVAL statement "source: = source.":".mvindex(split(ecs_task_definition,":"),0)" into component parts for update_source
Not sure what I'm doing wrong.
