Parsoning JSON arrays

drobMT — Tue, 20 Oct 2020 15:31:51 GMT

I'll start by saying I may be doing this completely wrong. 🙂

I need help removing the first 2 lines and the last 2 lines of a file via props and transforms. I have tried so far only works to remove the first to lines (so all events process properly except the last on in the file b/c the last 2 lines end up mucking up the json for that event).

I have a JSON file (sample content below); the file starts with "value" : [ with several hundred objects in the values in that array.

{ "value": [ { "properties": { "roleName": "Virtual Machine Administrator", "type": "CustomRole", "description": "administer and update virtual machines.", "assignableScopes": [ "/subscriptions/xxxxx-xxxx-xxxx-xxxx-xxxxxxxxx" ], "permissions": [ { "actions": [ "Microsoft.Storage/*/read", "Microsoft.Compute/virtualMachines/performMaintenance/action" ], "notActions": [] } ], "createdOn": "2018-11-01T20:32:29.71317Z", "updatedOn": "2018-11-01T20:32:29.71317Z", "createdBy": "af5e3f18-3a18-4141-8296-5efb1b267cd9", "updatedBy": "af5e3f18-3a18-4141-8296-5efb1b267cd9" }, "id": "/subscriptions/xxxxx-xxxx-xxxx-xxxx-xxxxxxxxx/providers/Microsoft.Authorization/roleDefinitions/92e07475-99a8-4e12-9fc2-c4034be97904", "type": "Microsoft.Authorization/roleDefinitions", "name": "xxxxx-xxxx-xxxx-xxxx-xxxxxxxxx" }, { "properties": { "roleName": "Virtual Machine Support", "type": "CustomRole", "description": "Can administer and update virtual machines.", "assignableScopes": [ "/subscriptions/xxxxx-xxxx-xxxx-xxxx-xxxxxxxxx", "/subscriptions/xxxxx-xxxx-xxxx-xxxx-xxxxxxxxx" ], "permissions": [ { "actions": [ "Microsoft.Storage/*/read", "Microsoft.Compute/disks/delete", "Microsoft.Compute/disks/write", "Microsoft.Compute/snapshots/write", "Microsoft.Compute/disks/beginGetAccess/action" ], "notActions": [] } ], "createdOn": "2018-11-28T02:09:47.2262816Z", "updatedOn": "2020-09-14T17:33:57.5619979Z", "createdBy": "xxxxx-xxxx-xxxx-xxxx-xxxxxxxxx", "updatedBy": "xxxxx-xxxx-xxxx-xxxx-xxxxxxxxx" }, "id": "/subscriptions/xxxxx-xxxx-xxxx-xxxx-xxxxxxxxx/providers/Microsoft.Authorization/roleDefinitions/e74f813f-9dee-48f4-a0ba-ec37f07a95f9", "type": "Microsoft.Authorization/roleDefinitions", "name": "xxxxx-xxxx-xxxx-xxxx-xxxxxxxxx" } ] }

All a care about is what is in the array (as individual events in splunk). So I'd like to is strip off (at the beginning)

{ "value": [

and remove the following from the end.

] }

If I do that then everything I have works perfectly for splunk. My current problem is that my props and transforms will remove { "value": [ from the beginning but I can't seem to remove the ] } from the end.

## props.conf [mscs:azure:roledef] TRANSFORMS-timestamp=timestampeval TRANSFORMS-elimL1=eliminateL1, eliminateLE KV_MODE = json LINE_BREAKER = (?ms)[\r\n]+\s{4}}(,[\n\r]+) NO_BINARY_CHECK = true SHOULD_LINEMERGE = false TRANSFORMS-timestamp = timestampeval TRUNCATE = 0 category = Structured description = A variant of the JSON source type, with support for nonexistent timestamps disabled = false pulldown_type = true ## transforms.conf [timestampeval] INGEST_EVAL = _time=strptime(replace(source,".*(?=\\\)\\\\",""),"Role Definitions_%Y-%m-%dT%H %M %S") [eliminateL1] REGEX = (?ms)^(?:{.+"value":\s\[.) DEST_KEY = queue FORMAT = nullQueue [eliminateLE] REGEX = (?ms)(?:\s+]\s})$ DEST_KEY = queue FORMAT = nullQueue

Re: Parsoning JSON arrays

drobMT — Fri, 23 Oct 2020 17:07:42 GMT

Well 💩 . After beating my head against the wall on this... I decided to clean the index on my dev box and try this again. It's working as expected and with the props and transforms above. I think I must have been looking at old events every time I was iterating and that was confusing me.

Leaving this here in case it can help someone else.

topic Parsoning JSON arrays in Getting Data In

Parsoning JSON arrays

Re: Parsoning JSON arrays