How to split JSON array element into different events using props.conf

[shanksholla]

Hi

I have a JSON message which looks like-

{
    "data": [
        {
            "id": "X999_Y999",
            "from": {
                "name": "Tom Brady",
                "id": "X12"
            },
            "message": "Looking forward to 2010!",
            "actions": [
                {
                    "name": "Comment",
                    "link": "http://www.facebook.com/X999/posts/Y999"
                },
                {
                    "name": "Like",
                    "link": "http://www.facebook.com/X999/posts/Y999"
                }
            ],
            "type": "status",
            "created_time": "2010-08-02T21:27:44+0000",
            "updated_time": "2010-08-02T21:27:44+0000"
        },
        {
            "id": "X998_Y998",
            "from": {
                "name": "Peyton Manning",
                "id": "X18"
            },
            "message": "Where's my contract?",
            "actions": [
                {
                    "name": "Comment",
                    "link": "http://www.facebook.com/X998/posts/Y998"
                },
                {
                    "name": "Like",
                    "link": "http://www.facebook.com/X998/posts/Y998"
                }
            ],
            "type": "status",
            "created_time": "2010-08-02T21:27:44+0000",
            "updated_time": "2010-08-02T21:27:44+0000"
        }
    ]
} 

Now, I would want to split this JSON array into different events. In this message, data from-
{
"id": "",
onwards would constitute different messages.

I have tried the following in props.conf, but all the data are shows up as single event.

[ json_split ]
TIME_FORMAT=%Y-%m-%dT%H:%M:%S+%4N
TIME_PREFIX="updated_time":\s"
SHOULD_LINEMERGE=true
BREAK_ONLY_BEFORE=(\{\s+"id")
NO_BINARY_CHECK=true

Can you please tell how to split this JSON array into different events.

[aljohnson_splun]

---

---

---

For your sample data, I changed

BREAK_ONLY_BEFORE="id":\s".*?",

and it worked - its not very robust however - if you ever have a field that comes after "id" in the "from" node, it will break.

[shanksholla]

Thanks for your reply!
But with this, the leading curly braces (prior to "id") is left in previous event.
Also with this, it wouldn't be possible to parse the data in events into key value pairs using INDEXED_EXTRACTIONS or KV_MODE.
Please correct me if I'm wrong

[aljohnson_splun]

Hmm. I see what your saying. I'm guessing it would be easier to either use LINE_BREAKER and turn SHOULD_LINEMERGE to false. I'll give it another go and see if I can come up with a better solution.

Could you maybe parse the events before indexing - it might be easier 🙂 ?

[shanksholla]

Can you please tell me how to parse before indexing. Is it using transforms.conf?