So I am using this stanza in my .../local props.conf
[mscs:storage:blob]
SHOULD_LINEMERGE = true
LINE_BREAKER: (,[\r\n]+\s+){
TRUNCATE = 0
KV_MODE = json
This works fine to parse my events like this
"time": "2018-07-09T16:14:21.3832528Z",
"resourceId": "blah",
"operationName": "blah blah",
"category": "blah",
"resultType": "blah",
"resultSignature": "blah",
"durationMs": 1234,
"callerIpAddress": "x.x.x.x",
"correlationId": "x-x-x-x-x",
"identity": {"blah blah":{"blah"}},
"level": "blah",
"location": "blah",
"properties": {"statusCode":"OK","serviceRequestId":"x-x-x"}
}
]
}
OR
"time": "2018-07-09T16:14:21.3832528Z",
"resourceId": "blah",
"operationName": "blah blah",
"category": "blah",
"resultType": "blah",
"resultSignature": "blah",
"durationMs": 1234,
"callerIpAddress": "x.x.x.x",
"correlationId": "x-x-x-x-x",
"identity": {"blah blah":{"blah"}},
"level": "blah",
"location": "blah",
"properties": {"statusCode":"OK","serviceRequestId":"x-x-x"}
}
,
{
Does anyone know how to drop the brackets and commas?
Thank you
For JSON in a blob, these props.conf setting work pretty well for most use cases:
[sourcetype]
LINE_BREAKER = \}([\r\n]\s*,[\r\n]\s*)\{
SEDCMD-remove_header = s/\{\s*\"records\"\:\s*\[\s*//g
SEDCMD-remove_footer = s/\][\r\n]\s*\}.*//g
SHOULD_LINEMERGE = false
KV_MODE = json
TIME_PREFIX = time\":\"
Thank you for the reply, I will give it a try and let you know.
Where is this from? This is pretty printed JSON data, it would be easier to fix the source to send JSON as single line events.
They are log files send to blob to be pulled into Splunk and ingested. Unfortunately I don't have access to the source to send in a perfect format.
Do you know a way to discard the brackets and commas?
Do you have a sample of the full log, not how Splunk parsed it?
We need to see how multiple events look.
Unfortunately I don't have access to the source logs, and probably would not be able to post the original without redacting a lot of sensitive info.
What I do know is that some of the events in the blob are Azure WAF logs.
Would you be able to use that as a source sample?
Here is a link that has a number of the sample json logs
https://docs.microsoft.com/en-us/azure/application-gateway/application-gateway-diagnostics
I appreciate your efforts.
thank you