Parsoning JSON arrays

[drobMT]

I'll start by saying I may be doing this completely wrong. 🙂

I need help removing the first 2 lines and the last 2 lines of a file via props and transforms. I have tried so far only works to remove the first to lines (so all events process properly except the last on in the file b/c the last 2 lines end up mucking up the json for that event).

I have a JSON file (sample content below); the file starts with "value" : [ with several hundred objects in the values in that array.

  {
  "value": [
    {
      "properties": {
        "roleName": "Virtual Machine Administrator",
        "type": "CustomRole",
        "description": "administer and update virtual machines.",
        "assignableScopes": [
          "/subscriptions/xxxxx-xxxx-xxxx-xxxx-xxxxxxxxx"
        ],
        "permissions": [
          {
            "actions": [
              "Microsoft.Storage/*/read",
              "Microsoft.Compute/virtualMachines/performMaintenance/action"
            ],
            "notActions": []
          }
        ],
        "createdOn": "2018-11-01T20:32:29.71317Z",
        "updatedOn": "2018-11-01T20:32:29.71317Z",
        "createdBy": "af5e3f18-3a18-4141-8296-5efb1b267cd9",
        "updatedBy": "af5e3f18-3a18-4141-8296-5efb1b267cd9"
      },
      "id": "/subscriptions/xxxxx-xxxx-xxxx-xxxx-xxxxxxxxx/providers/Microsoft.Authorization/roleDefinitions/92e07475-99a8-4e12-9fc2-c4034be97904",
      "type": "Microsoft.Authorization/roleDefinitions",
      "name": "xxxxx-xxxx-xxxx-xxxx-xxxxxxxxx"
    },
    {
      "properties": {
        "roleName": "Virtual Machine Support",
        "type": "CustomRole",
        "description": "Can administer and update virtual machines.",
        "assignableScopes": [
          "/subscriptions/xxxxx-xxxx-xxxx-xxxx-xxxxxxxxx",
          "/subscriptions/xxxxx-xxxx-xxxx-xxxx-xxxxxxxxx"
        ],
        "permissions": [
          {
            "actions": [
              "Microsoft.Storage/*/read",
              "Microsoft.Compute/disks/delete",
              "Microsoft.Compute/disks/write",
              "Microsoft.Compute/snapshots/write",
              "Microsoft.Compute/disks/beginGetAccess/action"
            ],
            "notActions": []
          }
        ],
        "createdOn": "2018-11-28T02:09:47.2262816Z",
        "updatedOn": "2020-09-14T17:33:57.5619979Z",
        "createdBy": "xxxxx-xxxx-xxxx-xxxx-xxxxxxxxx",
        "updatedBy": "xxxxx-xxxx-xxxx-xxxx-xxxxxxxxx"
      },
      "id": "/subscriptions/xxxxx-xxxx-xxxx-xxxx-xxxxxxxxx/providers/Microsoft.Authorization/roleDefinitions/e74f813f-9dee-48f4-a0ba-ec37f07a95f9",
      "type": "Microsoft.Authorization/roleDefinitions",
      "name": "xxxxx-xxxx-xxxx-xxxx-xxxxxxxxx"
    }
  ]
}

All a care about is what is in the array (as individual events in splunk). So I'd like to is strip off (at the beginning)

{
  "value": [

 and remove the following from the end.

  ]
}

If I do that then everything I have works perfectly for splunk. My current problem is that my props and transforms will remove { "value": [ from the beginning but I can't seem to remove the ] } from the end.

## props.conf
[mscs:azure:roledef]
TRANSFORMS-timestamp=timestampeval
TRANSFORMS-elimL1=eliminateL1, eliminateLE
KV_MODE = json
LINE_BREAKER = (?ms)[\r\n]+\s{4}}(,[\n\r]+)
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
TRANSFORMS-timestamp = timestampeval
TRUNCATE = 0
category = Structured
description = A variant of the JSON source type, with support for nonexistent timestamps
disabled = false
pulldown_type = true

## transforms.conf
[timestampeval]
INGEST_EVAL = _time=strptime(replace(source,".*(?=\\\)\\\\",""),"Role Definitions_%Y-%m-%dT%H %M %S")

[eliminateL1]
REGEX = (?ms)^(?:{.+"value":\s\[.)
DEST_KEY = queue
FORMAT = nullQueue

[eliminateLE]
REGEX = (?ms)(?:\s+]\s})$
DEST_KEY = queue
FORMAT = nullQueue

[drobMT · Accepted solution]

Well 💩 . After beating my head against the wall on this... I decided to clean the index on my dev box and try this again. It's working as expected and with the props and transforms above. I think I must have been looking at old events every time I was iterating and that was confusing me.

Leaving this here in case it can help someone else.