Splunk Enterprise Security

Azure: Is there a way Splunk, by default, extracts the fields from nested JSON logs?

yosplunksunny
New Member

Hi All,

Is there a way Splunk by default to extracts the fields from nested JSON logs? Right now Splunk is parsing the standard JSON files whoever it will not parse the value or ignore the nested JSON values however parse the rest of the data from the event.
Below is the sample event :
In this Splunk is parsing the data however field "policies":"["these values take as a single value rather parsing all other fields inside this field.

**{"time":"2019-09-23T10:52:06.6532250Z","resourceId":"/SUBSCRIPTIONS/111F6ACA-TC19-4567-H5M3-A8F8ECC49A58","operationName":"MICROSOFT.AUTHORIZATION/POLICIES/AUDIT/ACTION","category":"Action","resultType":"Success","resultSignature":"Succeeded.","durationMs":0,"callerIpAddress":"10.0.0.0","correlationId":"981cf31f-ffb2-415d-bbe4-j10d5637c393","identity":{"authorization":{"scope":"/subscriptions/111F6ACA-TC19-4567-H5M3-A8F8ECC49A58","action":"Microsoft.Resources/checkPolicyCompliance/read","evidence":{"role":"Policy Insights Resource Provider Service Role","roleAssignmentScope":"/subscriptions/111F6ACA-TC19-4567-H5M3-A8F8ECC49A58","roleAssignmentId":"30000e0edec8889988a018870ae4bdaf","roleDefinitionId":"30000e0edec8889688a018870ae4bdaf","principalId":"90111e0edec8889688a018870ae4bdaf","principalType":"ServicePrincipal"}},"claims":{"INR":"https://management.azure.com/","iss":"https://sts.windows.net/111F6ACA-TC19-4567-H5M3-A8F8ECC49A58/","iat":"1569235600","nbf":"5874","exp":"5656","aio":"10000+Y84Jsbb56565=","appid":"111F6ACA-TC19-4567-H5M3-A8F8ECC49A58","appidacr":"2","http://schemas.microsoft.com/identity/claims/identityprovider":"https://sts.windows.net/111F6ACA-TC19-4567-H5M3-A8F8ECC49A58/","http://schemas.microsoft.com/identity/claims/objectidentifier":"111F6ACA-TC19-4567-H5M3-A8F8ECC49A58","http://schemas.xmlsoap.org/ws/2005/05/identity/claims/nameidentifier":"111F6ACA-TC19-4567-H5M3-A8F8ECC49A58","http://schemas.microsoft.com/identity/claims/tenantid":"111F6ACA-TC19-4567-H5M3-A8F8ECC49A58","uti":"qbZDKv3YB0Six7cSv6eJFF","ver":"1.0"}},"level":"Warning","location":"global","properties":{"isComplianceCheck":"True","resourceLocation":null,"ancestors":"test-test-sandbox-non-production,test-non-production,test,111F6ACA-TC19-4567-H5M3-A8F8ECC49A58","policies":"[{\"policyDefinitionId\":\"/providers/Microsoft.Authorization/policyDefinitions/111F6ACA-TC19-4567-H5M3-A8F8ECC49A58/\",\"policySetDefinitionId\":\"/providers/Microsoft.Authorization/policySetDefinitions/111F6ACA-TC19-4567-H5M3-A8F8ECC49A58/\",\"policyDefinitionReferenceId\":\"identityRemoveExternalAccountWithWritePermissionsMonitoring\",\"policySetDefinitionName\":\"111F6ACA-TC19-4567-H5M3-A8F8ECC49A58\",\"policyDefinitionName\":\"111F6ACA-TC19-4567-H5M3-A8F8ECC49A58\",\"policyDefinitionEffect\":\"AuditIfNotExists\",\"policyAssignmentId\":\"/subscriptions/111F6ACA-TC19-4567-H5M3-A8F8ECC49A58/providers/Microsoft.Authorization/policyAssignments/SecurityCenterBuiltIn/\",\"policyAssignmentName\":\"SecurityCenterBuiltIn\",\"policyAssignmentScope\":\"/subscriptions/111F6ACA-TC19-4567-H5M3-A8F8ECC49A58\",\"policyAssignmentSku\":{\"name\":\"A1\",\"tier\":\"Standard\"},\"policyAssignmentParameters\":{}}]"},"am_category":"ACTION"}**

By default search time or index time , can we extract this nested values rather than using spath to extract in search.

As we need to use this is in the data model values once we get this field extracted.

Thanks in advance.

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

Have you looked at this answer from the archives? Essentially, the escaping is removed from the escaped double quotes, and the embedded field is unquoted, before parsing with spath, allowing the whole log to be parsed as you were hoping for.

0 Karma

snaikwade1
Observer

Hi,

I am more so looking for extracting these fields and them making them CIM compliant. The microsoft cloud services add-on does not help much as the fields are not extracted and data model mapping is not done correctly. 

0 Karma

to4kawa
Ultra Champion

props.conf
[your sourcetype]
INDEXED_EXTRACTIONS = json
TRANSFORMS-azure_sample

transforms.conf
[azure_sample]
SOURCE_KEY=properties.policies
REGEX = \"(?<name>[^\"]+)\":\"(?<value>[^\"]+)\"
FORMAT = $1::"$2"
REPEAT_MATCH = true
WRITE_META = true

I haven't tried it, but how about this?

0 Karma

snaikwade1
Observer

Hi,

were you able to resolve the issue ?

0 Karma

eltra1n
New Member

Looking in to this as well, will post results. Do we know why  events are treated differently?

0 Karma
Get Updates on the Splunk Community!

Optimize Cloud Monitoring

  TECH TALKS Optimize Cloud Monitoring Tuesday, August 13, 2024  |  11:00AM–12:00PM PST   Register to ...

What's New in Splunk Cloud Platform 9.2.2403?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.2.2403! Analysts can ...

Stay Connected: Your Guide to July and August Tech Talks, Office Hours, and Webinars!

Dive into our sizzling summer lineup for July and August Community Office Hours and Tech Talks. Scroll down to ...