How to spath or rename commands not extracting field / values?

[bt149]

I have a field called properties.requestbody.  I would like to have this field broken out based on the field and values paired.  I've tried with spath and no luck.  I've used and am using rename to extract the field / values in other parts of the logged events.  Not having luck with this field.  I think it has to do with the quotes but I'm not certain.  Thanks as always for the help and guidance.

"properties": {"requestbody": "{\"properties\":{\"description\":\"Azure Private Link lets you connect your virtual network to Azure services without a public IP address at the source or destination. The Private Link platform handles the connectivity between the consumer and services over the Azure backbone network. By mapping private endpoints to diskAccesses, data leakage risks are reduced. Learn more about private links at: https://aka.ms/disksprivatelinksdoc. \",\"displayName\":\"COMP-015N-Disk access resources should use private link-AuditIfNotExists-BUL\",\"metadata\":\"******\",\"mode\":\"Indexed\",\"parameters\":\"******\",\"policyRule\":\"******\",\"policyType\":\"Custom\"}}"

[yuanliu]

I am a little confused.  Your actual events should look like this:

{"properties": {"requestbody": "{\"properties\":{\"description\":\"Azure Private Link lets you connect your virtual network to Azure services without a public IP address at the source or destination. The Private Link platform handles the connectivity between the consumer and services over the Azure backbone network. By mapping private endpoints to diskAccesses, data leakage risks are reduced. Learn more about private links at: https://aka.ms/disksprivatelinksdoc. \",\"displayName\":\"COMP-015N-Disk access resources should use private link-AuditIfNotExists-BUL\",\"metadata\":\"******\",\"mode\":\"Indexed\",\"parameters\":\"******\",\"policyRule\":\"******\",\"policyType\":\"Custom\"}}"}}

Is this about right? (If the raw data is not conformant JSON, you can try to make it conformant, then use spath.) Splunk already gives you a field properties.requestbody, with this value:

{"properties":{"description":"Azure Private Link lets you connect your virtual network to Azure services without a public IP address at the source or destination. The Private Link platform handles the connectivity between the consumer and services over the Azure backbone network. By mapping private endpoints to diskAccesses, data leakage risks are reduced. Learn more about private links at: https://aka.ms/disksprivatelinksdoc. ","displayName":"COMP-015N-Disk access resources should use private link-AuditIfNotExists-BUL","metadata":"******","mode":"Indexed","parameters":"******","policyRule":"******","policyType":"Custom"}}

If the above is correct, all you need to do is to run spath against this field, i.e.,

| spath input=properties.requestbody

Using this, your sample data gives

properties.description	properties.displayName	properties.metadata	properties.mode	properties.parameters	properties.policyRule	properties.policyType
Azure Private Link lets you connect your virtual network to Azure services without a public IP address at the source or destination. The Private Link platform handles the connectivity between the consumer and services over the Azure backbone network. By mapping private endpoints to diskAccesses, data leakage risks are reduced. Learn more about private links at: https://aka.ms/disksprivatelinksdoc.	COMP-015N-Disk access resources should use private link-AuditIfNotExists-BUL	******	Indexed	******	******	Custom

Emulation

This is the emulation I use to test

| makeresults
| fields - _time
| eval _raw = "{\"properties\": {\"requestbody\": \"{\\\"properties\\\":{\\\"description\\\":\\\"Azure Private Link lets you connect your virtual network to Azure services without a public IP address at the source or destination. The Private Link platform handles the connectivity between the consumer and services over the Azure backbone network. By mapping private endpoints to diskAccesses, data leakage risks are reduced. Learn more about private links at: https://aka.ms/disksprivatelinksdoc. \\\",\\\"displayName\\\":\\\"COMP-015N-Disk access resources should use private link-AuditIfNotExists-BUL\\\",\\\"metadata\\\":\\\"******\\\",\\\"mode\\\":\\\"Indexed\\\",\\\"parameters\\\":\\\"******\\\",\\\"policyRule\\\":\\\"******\\\",\\\"policyType\\\":\\\"Custom\\\"}}\"}}"
``` data emulation above ```
| spath

[bt149]

Don't be confused, my events look like how I posted and I believe that along w/ the quotes are causing the issue.

[yuanliu]

Your original post says "I have a field called properties.requestbody."  That is only possible if the event is actually what I mended.

Now, if the original event is indeed missing the left open curly bracket, and missing two closing missing brackets, you can "mend" it by simply adding them.

| eval _raw = replace(_raw, "(.*)", "{\1}}")
| spath
| spath input=properties.requestbody

Using SPL's builtin command would be much more robust than using rex to handle structured data.

Emulation

Here is an emulation to demonstrate:

| makeresults
| fields - _time
| eval _raw = "\"properties\": {\"requestbody\": \"{\\\"properties\\\":{\\\"description\\\":\\\"Azure Private Link lets you connect your virtual network to Azure services without a public IP address at the source or destination. The Private Link platform handles the connectivity between the consumer and services over the Azure backbone network. By mapping private endpoints to diskAccesses, data leakage risks are reduced. Learn more about private links at: https://aka.ms/disksprivatelinksdoc. \\\",\\\"displayName\\\":\\\"COMP-015N-Disk access resources should use private link-AuditIfNotExists-BUL\\\",\\\"metadata\\\":\\\"******\\\",\\\"mode\\\":\\\"Indexed\\\",\\\"parameters\\\":\\\"******\\\",\\\"policyRule\\\":\\\"******\\\",\\\"policyType\\\":\\\"Custom\\\"}}\""
``` data emulation above ```

It gives the exact string in your OP

_raw

"properties": {"requestbody": "{\"properties\":{\"description\":\"Azure Private Link lets you connect your virtual network to Azure services without a public IP address at the source or destination. The Private Link platform handles the connectivity between the consumer and services over the Azure backbone network. By mapping private endpoints to diskAccesses, data leakage risks are reduced. Learn more about private links at: https://aka.ms/disksprivatelinksdoc. \",\"displayName\":\"COMP-015N-Disk access resources should use private link-AuditIfNotExists-BUL\",\"metadata\":\"******\",\"mode\":\"Indexed\",\"parameters\":\"******\",\"policyRule\":\"******\",\"policyType\":\"Custom\"}}"

Use replacement on it,

| makeresults
| fields - _time
| eval _raw = "\"properties\": {\"requestbody\": \"{\\\"properties\\\":{\\\"description\\\":\\\"Azure Private Link lets you connect your virtual network to Azure services without a public IP address at the source or destination. The Private Link platform handles the connectivity between the consumer and services over the Azure backbone network. By mapping private endpoints to diskAccesses, data leakage risks are reduced. Learn more about private links at: https://aka.ms/disksprivatelinksdoc. \\\",\\\"displayName\\\":\\\"COMP-015N-Disk access resources should use private link-AuditIfNotExists-BUL\\\",\\\"metadata\\\":\\\"******\\\",\\\"mode\\\":\\\"Indexed\\\",\\\"parameters\\\":\\\"******\\\",\\\"policyRule\\\":\\\"******\\\",\\\"policyType\\\":\\\"Custom\\\"}}\""
``` data emulation above ```
| eval _raw = replace(_raw, "(.*)", "{\1}}")

Then, you get the exact conformant JSON in my previous reply.

_raw

{"properties": {"requestbody": "{\"properties\":{\"description\":\"Azure Private Link lets you connect your virtual network to Azure services without a public IP address at the source or destination. The Private Link platform handles the connectivity between the consumer and services over the Azure backbone network. By mapping private endpoints to diskAccesses, data leakage risks are reduced. Learn more about private links at: https://aka.ms/disksprivatelinksdoc. \",\"displayName\":\"COMP-015N-Disk access resources should use private link-AuditIfNotExists-BUL\",\"metadata\":\"******\",\"mode\":\"Indexed\",\"parameters\":\"******\",\"policyRule\":\"******\",\"policyType\":\"Custom\"}}"}}

[richgalloway]

Splunk is fully capable of handling quotation marks in JSON events.  I believe we've demonstrated that.  The issue (as I see it) is the sample event is not valid JSON and the spath command will not process it.  AFAIK, none of the other JSON commands will work with invalid JSON, either.

[bt149]

I ended up using rex to get done what needed to be done.  Thanks for the help and guidance.

[richgalloway]

The spath command won't work with anything less than perfect JSON.  You can see this in the search log from the message "

 WARN  SPathCommand [22744 phase_1] - Some events are not in XML or JSON format.  Fields will not be extracted from these events.

 Adding {} around the event helped.

BTW, the rename command does not extract fields.  It merely changes the names of existing fields.

[bt149]

Thanks for the reply.  I don't see the WARN you posted.  I'm not expecting the "rename" cmd to do field extractions.

In these logged events there is another set of fields called identity.authorization.action and identity.authorization.scop.  Using the rename command I then get  two fields w/ corresponding values.

| rename identity.authorization.* AS *

     will produce action = "value"
     will produce scope = "value"

The difference is in the "identity" field the action and scope don't have quotes around the values where as the "properties.requestbody" field has values around all the fields and values.  Hope that makes sense.

[richgalloway]

The fields without quoted values will need a different regex.  How different depends on what values are expected and how Splunk can know where the value ends.  It may be possible to craft a regex that extracts either quoted or unquoted values, but that still means knowing where the value ends.

[bt149]

The fields w/out quotes I have no issues with, just the one with the quotes.