Splunk Enterprise Security

Azure: Is there a way Splunk, by default, extracts the fields from nested JSON logs?

yosplunksunny
New Member

Hi All,

Is there a way Splunk by default to extracts the fields from nested JSON logs? Right now Splunk is parsing the standard JSON files whoever it will not parse the value or ignore the nested JSON values however parse the rest of the data from the event.
Below is the sample event :
In this Splunk is parsing the data however field "policies":"["these values take as a single value rather parsing all other fields inside this field.

**{"time":"2019-09-23T10:52:06.6532250Z","resourceId":"/SUBSCRIPTIONS/111F6ACA-TC19-4567-H5M3-A8F8ECC49A58","operationName":"MICROSOFT.AUTHORIZATION/POLICIES/AUDIT/ACTION","category":"Action","resultType":"Success","resultSignature":"Succeeded.","durationMs":0,"callerIpAddress":"10.0.0.0","correlationId":"981cf31f-ffb2-415d-bbe4-j10d5637c393","identity":{"authorization":{"scope":"/subscriptions/111F6ACA-TC19-4567-H5M3-A8F8ECC49A58","action":"Microsoft.Resources/checkPolicyCompliance/read","evidence":{"role":"Policy Insights Resource Provider Service Role","roleAssignmentScope":"/subscriptions/111F6ACA-TC19-4567-H5M3-A8F8ECC49A58","roleAssignmentId":"30000e0edec8889988a018870ae4bdaf","roleDefinitionId":"30000e0edec8889688a018870ae4bdaf","principalId":"90111e0edec8889688a018870ae4bdaf","principalType":"ServicePrincipal"}},"claims":{"INR":"https://management.azure.com/","iss":"https://sts.windows.net/111F6ACA-TC19-4567-H5M3-A8F8ECC49A58/","iat":"1569235600","nbf":"5874","exp":"5656","aio":"10000+Y84Jsbb56565=","appid":"111F6ACA-TC19-4567-H5M3-A8F8ECC49A58","appidacr":"2","http://schemas.microsoft.com/identity/claims/identityprovider":"https://sts.windows.net/111F6ACA-TC19-4567-H5M3-A8F8ECC49A58/","http://schemas.microsoft.com/identity/claims/objectidentifier":"111F6ACA-TC19-4567-H5M3-A8F8ECC49A58","http://schemas.xmlsoap.org/ws/2005/05/identity/claims/nameidentifier":"111F6ACA-TC19-4567-H5M3-A8F8ECC49A58","http://schemas.microsoft.com/identity/claims/tenantid":"111F6ACA-TC19-4567-H5M3-A8F8ECC49A58","uti":"qbZDKv3YB0Six7cSv6eJFF","ver":"1.0"}},"level":"Warning","location":"global","properties":{"isComplianceCheck":"True","resourceLocation":null,"ancestors":"test-test-sandbox-non-production,test-non-production,test,111F6ACA-TC19-4567-H5M3-A8F8ECC49A58","policies":"[{\"policyDefinitionId\":\"/providers/Microsoft.Authorization/policyDefinitions/111F6ACA-TC19-4567-H5M3-A8F8ECC49A58/\",\"policySetDefinitionId\":\"/providers/Microsoft.Authorization/policySetDefinitions/111F6ACA-TC19-4567-H5M3-A8F8ECC49A58/\",\"policyDefinitionReferenceId\":\"identityRemoveExternalAccountWithWritePermissionsMonitoring\",\"policySetDefinitionName\":\"111F6ACA-TC19-4567-H5M3-A8F8ECC49A58\",\"policyDefinitionName\":\"111F6ACA-TC19-4567-H5M3-A8F8ECC49A58\",\"policyDefinitionEffect\":\"AuditIfNotExists\",\"policyAssignmentId\":\"/subscriptions/111F6ACA-TC19-4567-H5M3-A8F8ECC49A58/providers/Microsoft.Authorization/policyAssignments/SecurityCenterBuiltIn/\",\"policyAssignmentName\":\"SecurityCenterBuiltIn\",\"policyAssignmentScope\":\"/subscriptions/111F6ACA-TC19-4567-H5M3-A8F8ECC49A58\",\"policyAssignmentSku\":{\"name\":\"A1\",\"tier\":\"Standard\"},\"policyAssignmentParameters\":{}}]"},"am_category":"ACTION"}**

By default search time or index time , can we extract this nested values rather than using spath to extract in search.

As we need to use this is in the data model values once we get this field extracted.

Thanks in advance.

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

Have you looked at this answer from the archives? Essentially, the escaping is removed from the escaped double quotes, and the embedded field is unquoted, before parsing with spath, allowing the whole log to be parsed as you were hoping for.

0 Karma

snaikwade1
Observer

Hi,

I am more so looking for extracting these fields and them making them CIM compliant. The microsoft cloud services add-on does not help much as the fields are not extracted and data model mapping is not done correctly. 

0 Karma

to4kawa
Ultra Champion

props.conf
[your sourcetype]
INDEXED_EXTRACTIONS = json
TRANSFORMS-azure_sample

transforms.conf
[azure_sample]
SOURCE_KEY=properties.policies
REGEX = \"(?<name>[^\"]+)\":\"(?<value>[^\"]+)\"
FORMAT = $1::"$2"
REPEAT_MATCH = true
WRITE_META = true

I haven't tried it, but how about this?

0 Karma

snaikwade1
Observer

Hi,

were you able to resolve the issue ?

0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!