Extract aws service name from source field of metadata

[Poojitha]

Hi All,

I want to extract service name from sourcetype="aws:metadata" and source field.

Example : 434531263412:eu-central-1:elasticache_describe_reserved_cache_nodes_offerings

I am using this query :

index=* sourcetype=aws:metadata
| eval aws_service=mvindex(split(source,":"),2)
| rex field=aws_service "(?<aws_service>[^_]+)"
| table aws_service  source| dedup aws_service

Using this I will get result :  elasticache. But in case of "434531263412:us-west-2:nat_gateways" its just extracting nat. But it should be gateways. S

Similarly in 434531263412:eu-central-1:application_load_balancers, its extracting application.

I was thinking if we can check for the keyword and update the value. I want to add this in props.conf file so aws_service field gets created from source.

Please can anyone of you help me how can I achieve this ?

Regards,
PNV

[Poojitha]

@richgalloway  : If dont use rex , it gets entire value i.e nat_gateways. I just want nat.

My requirement is it should just extract service name.

Example :
434531263412:us-west-2:lambda_functions it will be lambda_functions. This is straight forward.
But like in  : "434531263412:us-west-2:nat_gateways",  it should be gateways. 
434531263412:us-west-2:application_load_balancers,  it should be load_balancers

This is my requirement. 

[richgalloway]

To summarize:

434531263412:us-west-2:lambda_functions -> lambda_functions
434531263412:us-west-2:nat_gateways -> gateways
434531263412:us-west-2:application_load_balancers -> load_balancers

If this is correct then more information is needed.  What is the rule to use to determine how much of the service is to be used?

[Poojitha]

@richgalloway  : Sorry I did not get what rule you are mentioning. Could you please be more clear on this ?

434531263412:us-west-2:lambda_functions -> lambda_functions
434531263412:us-west-2:nat_gateways -> gateways
434531263412:us-west-2:application_load_balancers -> load_balancers

yes , this is the requirement. In the above , right side values are the values from source field. I want to extract service name from this field value.

[richgalloway]

The requirements are inconsistent.  Sometimes everything after the second : is the service name; other times the service name follows the first _.  How is a computer to decide which method to use?

[Poojitha]

@richgalloway  I agree to your point. I tried using case statement as well . Unfortunately its not working as expected. Do you know any other way to handle this ? That really helps me. I am also re-searching.

[richgalloway]

I can't help if I don't understand what the goal is.  Once we have a deterministic way to set the service name I may be able to help.

[richgalloway]

The split function is extracting the desired field, but then rex reduces it to the part before the first underscore (_).  Remove the rex command and the query should work as expected.

In props..conf, add a transform that uses INGEST_EVAL

INGEST_EVAL = aws_service=mvindex(split(source,":"),2)