topic Re: How to split JSON array element into different events using props.conf in Getting Data In

How to split JSON array element into different events using props.conf

shanksholla — Tue, 25 Aug 2015 12:34:00 GMT

I have a JSON message which looks like-

{
    "data": [
        {
            "id": "X999_Y999",
            "from": {
                "name": "Tom Brady",
                "id": "X12"
            },
            "message": "Looking forward to 2010!",
            "actions": [
                {
                    "name": "Comment",
                    "link": "http://www.facebook.com/X999/posts/Y999"
                },
                {
                    "name": "Like",
                    "link": "http://www.facebook.com/X999/posts/Y999"
                }
            ],
            "type": "status",
            "created_time": "2010-08-02T21:27:44+0000",
            "updated_time": "2010-08-02T21:27:44+0000"
        },
        {
            "id": "X998_Y998",
            "from": {
                "name": "Peyton Manning",
                "id": "X18"
            },
            "message": "Where's my contract?",
            "actions": [
                {
                    "name": "Comment",
                    "link": "http://www.facebook.com/X998/posts/Y998"
                },
                {
                    "name": "Like",
                    "link": "http://www.facebook.com/X998/posts/Y998"
                }
            ],
            "type": "status",
            "created_time": "2010-08-02T21:27:44+0000",
            "updated_time": "2010-08-02T21:27:44+0000"
        }
    ]
}

Now, I would want to split this JSON array into different events. In this message, data from-
{ "id": "",
onwards would constitute different messages.

I have tried the following in props.conf, but all the data are shows up as single event.

[ json_split ]
TIME_FORMAT=%Y-%m-%dT%H:%M:%S+%4N
TIME_PREFIX="updated_time":\s"
SHOULD_LINEMERGE=true
BREAK_ONLY_BEFORE=(\{\s+"id")
NO_BINARY_CHECK=true

Can you please tell how to split this JSON array into different events.

Re: How to split JSON array element into different events using props.conf

aljohnson_splun — Tue, 25 Aug 2015 12:50:01 GMT

For your sample data, I changed

BREAK_ONLY_BEFORE="id":\s".*?",

and it worked - its not very robust however - if you ever have a field that comes after "id" in the "from" node, it will break.

Re: How to split JSON array element into different events using props.conf

shanksholla — Tue, 29 Sep 2020 07:07:43 GMT

Thanks for your reply!
But with this, the leading curly braces (prior to "id") is left in previous event.
Also with this, it wouldn't be possible to parse the data in events into key value pairs using INDEXED_EXTRACTIONS or KV_MODE.
Please correct me if I'm wrong

Re: How to split JSON array element into different events using props.conf

aljohnson_splun — Tue, 29 Sep 2020 07:04:46 GMT

Hmm. I see what your saying. I'm guessing it would be easier to either use LINE_BREAKER and turn SHOULD_LINEMERGE to false. I'll give it another go and see if I can come up with a better solution.

Could you maybe parse the events before indexing - it might be easier 🙂 ?

Re: How to split JSON array element into different events using props.conf

shanksholla — Thu, 27 Aug 2015 11:33:38 GMT

Can you please tell me how to parse before indexing. Is it using transforms.conf?