Splunk Cloud: Normalize subscriptionId to lowercase

[Leonardo1998]

Hi everyone,

I'm using Splunk Cloud with the Splunk Add-on for Microsoft Cloud Services  to manage two Azure subscriptions. As a result, I have duplicated inputs, and I need a way to reference each subscription within my queries.

I noticed that the subscriptionId field exists, but it contains four variations: two in lowercase and two in uppercase. I'd like to normalize this field to lowercase at ingest time, so I don't have to handle it manually in every query.

I checked the Field Transformations, but I couldn't find any mention of subscriptionId (I only see subscription_id).

Has anyone dealt with a similar issue, or can anyone suggest the best approach?
Thanks in advance for your help!

(P.S. I'm relatively new to Splunk and Splunk Cloud, so any guidance is greatly appreciated!)

[livehybrid]

Hi @Leonardo1998 

In order to index this as a lowercase field, we need to establish how its derived. 

Checking the app's props/transforms there are a number of REGEX which extract "subscription_id" from various fields.

such as below, however like you mentioned - this are subscription_id not subscriptionId!

[mscs_extract_subscription_id_and_resource_group]
SOURCE_KEY = AzureResourceId
REGEX = (?i:subscriptions)\/([^\/]+)(?:\/(?i:resourceGroups)\/([^\/]+))?
FORMAT = subscription_id::$1 resource_group::$2

[mscs_extract_subscription_id_and_resource_group_from_id]
SOURCE_KEY = id
REGEX = (?i:subscriptions)\/([^\/]+)(?:\/(?i:resourceGroups)\/([^\/]+))?
FORMAT = subscription_id::$1 resource_group::$2

However.. I did find this:

[azure_data_share_extract_from_properties]
SOURCE_KEY = properties
REGEX = \"(\w+)\":\"({.*}|.*?)\"
FORMAT = $1::$2

Which extracts keyvalue pairs from properties and I *think* subscriptionId and subscriptionid get extracted from, based on this:

coalesce('subscriptionId', 'properties.subscriptionId', 'properties.subscriptionid', SUBSCRIPTIONS)

It looks like the source data contains different cased fields...not ideal!

Anyway - If you let me know the sourcetype you are looking at I can try and help put together an index-time props/transforms to index this...or...the other thing you might like to do is an eval field to coalesce them at search-time so you have a consistent value. You might actually find that "vendor_account" already does this, but if not you could do this:

[yourSourcetype]
EVAL-subscriptionId=COALESCE(subscriptionId,subscriptionid)

However would need to check the order of execution for the EVAL - or just see if it works 😉

Please let me know how you get on and consider adding karma to this or any other answer if it has helped.
Regards

Will