Azure: Is there a way Splunk, by default, extracts the fields from nested JSON logs?

[yosplunksunny]

Hi All,

Is there a way Splunk by default to extracts the fields from nested JSON logs? Right now Splunk is parsing the standard JSON files whoever it will not parse the value or ignore the nested JSON values however parse the rest of the data from the event.
Below is the sample event :
In this Splunk is parsing the data however field "policies":"["these values take as a single value rather parsing all other fields inside this field.

**{"time":"2019-09-23T10:52:06.6532250Z","resourceId":"/SUBSCRIPTIONS/111F6ACA-TC19-4567-H5M3-A8F8ECC49A58","operationName":"MICROSOFT.AUTHORIZATION/POLICIES/AUDIT/ACTION","category":"Action","resultType":"Success","resultSignature":"Succeeded.","durationMs":0,"callerIpAddress":"10.0.0.0","correlationId":"981cf31f-ffb2-415d-bbe4-j10d5637c393","identity":{"authorization":{"scope":"/subscriptions/111F6ACA-TC19-4567-H5M3-A8F8ECC49A58","action":"Microsoft.Resources/checkPolicyCompliance/read","evidence":{"role":"Policy Insights Resource Provider Service Role","roleAssignmentScope":"/subscriptions/111F6ACA-TC19-4567-H5M3-A8F8ECC49A58","roleAssignmentId":"30000e0edec8889988a018870ae4bdaf","roleDefinitionId":"30000e0edec8889688a018870ae4bdaf","principalId":"90111e0edec8889688a018870ae4bdaf","principalType":"ServicePrincipal"}},"claims":{"INR":"https://management.azure.com/","iss":"https://sts.windows.net/111F6ACA-TC19-4567-H5M3-A8F8ECC49A58/","iat":"1569235600","nbf":"5874","exp":"5656","aio":"10000+Y84Jsbb56565=","appid":"111F6ACA-TC19-4567-H5M3-A8F8ECC49A58","appidacr":"2","http://schemas.microsoft.com/identity/claims/identityprovider":"https://sts.windows.net/111F6ACA-TC19-4567-H5M3-A8F8ECC49A58/","http://schemas.microsoft.com/identity/claims/objectidentifier":"111F6ACA-TC19-4567-H5M3-A8F8ECC49A58","http://schemas.xmlsoap.org/ws/2005/05/identity/claims/nameidentifier":"111F6ACA-TC19-4567-H5M3-A8F8ECC49A58","http://schemas.microsoft.com/identity/claims/tenantid":"111F6ACA-TC19-4567-H5M3-A8F8ECC49A58","uti":"qbZDKv3YB0Six7cSv6eJFF","ver":"1.0"}},"level":"Warning","location":"global","properties":{"isComplianceCheck":"True","resourceLocation":null,"ancestors":"test-test-sandbox-non-production,test-non-production,test,111F6ACA-TC19-4567-H5M3-A8F8ECC49A58","policies":"[{\"policyDefinitionId\":\"/providers/Microsoft.Authorization/policyDefinitions/111F6ACA-TC19-4567-H5M3-A8F8ECC49A58/\",\"policySetDefinitionId\":\"/providers/Microsoft.Authorization/policySetDefinitions/111F6ACA-TC19-4567-H5M3-A8F8ECC49A58/\",\"policyDefinitionReferenceId\":\"identityRemoveExternalAccountWithWritePermissionsMonitoring\",\"policySetDefinitionName\":\"111F6ACA-TC19-4567-H5M3-A8F8ECC49A58\",\"policyDefinitionName\":\"111F6ACA-TC19-4567-H5M3-A8F8ECC49A58\",\"policyDefinitionEffect\":\"AuditIfNotExists\",\"policyAssignmentId\":\"/subscriptions/111F6ACA-TC19-4567-H5M3-A8F8ECC49A58/providers/Microsoft.Authorization/policyAssignments/SecurityCenterBuiltIn/\",\"policyAssignmentName\":\"SecurityCenterBuiltIn\",\"policyAssignmentScope\":\"/subscriptions/111F6ACA-TC19-4567-H5M3-A8F8ECC49A58\",\"policyAssignmentSku\":{\"name\":\"A1\",\"tier\":\"Standard\"},\"policyAssignmentParameters\":{}}]"},"am_category":"ACTION"}**

By default search time or index time , can we extract this nested values rather than using spath to extract in search.

As we need to use this is in the data model values once we get this field extracted.

Thanks in advance.

[ITWhisperer]

Have you looked at this answer from the archives? Essentially, the escaping is removed from the escaped double quotes, and the embedded field is unquoted, before parsing with spath, allowing the whole log to be parsed as you were hoping for.

[snaikwade1]

Hi,

I am more so looking for extracting these fields and them making them CIM compliant. The microsoft cloud services add-on does not help much as the fields are not extracted and data model mapping is not done correctly. 

[to4kawa]

props.conf
[your sourcetype]
INDEXED_EXTRACTIONS = json
TRANSFORMS-azure_sample

transforms.conf
[azure_sample]
SOURCE_KEY=properties.policies
REGEX = \"(?<name>[^\"]+)\":\"(?<value>[^\"]+)\"
FORMAT = $1::"$2"
REPEAT_MATCH = true
WRITE_META = true

I haven't tried it, but how about this?

[snaikwade1]

Hi,

were you able to resolve the issue ?

[eltra1n]

Looking in to this as well, will post results. Do we know why  events are treated differently?