topic Re: Extract aws service name from source field of metadata in Splunk Search

Extract aws service name from source field of metadata

Poojitha — Thu, 18 Apr 2024 13:10:40 GMT

Hi All,

I want to extract service name from sourcetype="aws:metadata" and source field.

Example : 434531263412:eu-central-1:elasticache_describe_reserved_cache_nodes_offerings

I am using this query :

index=* sourcetype=aws:metadata | eval aws_service=mvindex(split(source,":"),2) | rex field=aws_service "(?<aws_service>[^_]+)" | table aws_service source| dedup aws_service

Using this I will get result : elasticache. But in case of "434531263412:us-west-2:nat_gateways" its just extracting nat. But it should be gateways. S

Similarly in 434531263412:eu-central-1:application_load_balancers, its extracting application.

I was thinking if we can check for the keyword and update the value. I want to add this in props.conf file so aws_service field gets created from source.

Please can anyone of you help me how can I achieve this ?

Regards,
PNV

Re: Extract aws service name from source field of metadata

richgalloway — Thu, 18 Apr 2024 13:21:00 GMT

The split function is extracting the desired field, but then rex reduces it to the part before the first underscore (_). Remove the rex command and the query should work as expected.

In props..conf, add a transform that uses INGEST_EVAL

INGEST_EVAL = aws_service=mvindex(split(source,":"),2)

Re: Extract aws service name from source field of metadata

Poojitha — Thu, 18 Apr 2024 13:55:00 GMT

@richgalloway : If dont use rex , it gets entire value i.e nat_gateways. I just want nat.

My requirement is it should just extract service name.

Example :
434531263412:us-west-2:lambda_functions it will be lambda_functions. This is straight forward.
But like in : "434531263412:us-west-2:nat_gateways", it should be gateways.
434531263412:us-west-2:application_load_balancers, it should be load_balancers

This is my requirement.

Re: Extract aws service name from source field of metadata

richgalloway — Thu, 18 Apr 2024 17:00:19 GMT

To summarize:

434531263412:us-west-2:lambda_functions -> lambda_functions
434531263412:us-west-2:nat_gateways -> gateways
434531263412:us-west-2:application_load_balancers -> load_balancers

If this is correct then more information is needed. What is the rule to use to determine how much of the service is to be used?

Re: Extract aws service name from source field of metadata

Poojitha — Thu, 18 Apr 2024 18:15:18 GMT

@richgalloway : Sorry I did not get what rule you are mentioning. Could you please be more clear on this ?

434531263412:us-west-2:lambda_functions -> lambda_functions
434531263412:us-west-2:nat_gateways -> gateways
434531263412:us-west-2:application_load_balancers -> load_balancers

yes , this is the requirement. In the above , right side values are the values from source field. I want to extract service name from this field value.

Re: Extract aws service name from source field of metadata

richgalloway — Thu, 18 Apr 2024 20:32:12 GMT

The requirements are inconsistent. Sometimes everything after the second : is the service name; other times the service name follows the first _. How is a computer to decide which method to use?

Re: Extract aws service name from source field of metadata

Poojitha — Fri, 19 Apr 2024 06:37:30 GMT

@richgalloway I agree to your point. I tried using case statement as well . Unfortunately its not working as expected. Do you know any other way to handle this ? That really helps me. I am also re-searching.

Re: Extract aws service name from source field of metadata

richgalloway — Fri, 19 Apr 2024 12:07:47 GMT

I can't help if I don't understand what the goal is. Once we have a deterministic way to set the service name I may be able to help.