Getting Data In

How to split JSON array element into different events using props.conf

shanksholla
Explorer

Hi

I have a JSON message which looks like-

{
    "data": [
        {
            "id": "X999_Y999",
            "from": {
                "name": "Tom Brady",
                "id": "X12"
            },
            "message": "Looking forward to 2010!",
            "actions": [
                {
                    "name": "Comment",
                    "link": "http://www.facebook.com/X999/posts/Y999"
                },
                {
                    "name": "Like",
                    "link": "http://www.facebook.com/X999/posts/Y999"
                }
            ],
            "type": "status",
            "created_time": "2010-08-02T21:27:44+0000",
            "updated_time": "2010-08-02T21:27:44+0000"
        },
        {
            "id": "X998_Y998",
            "from": {
                "name": "Peyton Manning",
                "id": "X18"
            },
            "message": "Where's my contract?",
            "actions": [
                {
                    "name": "Comment",
                    "link": "http://www.facebook.com/X998/posts/Y998"
                },
                {
                    "name": "Like",
                    "link": "http://www.facebook.com/X998/posts/Y998"
                }
            ],
            "type": "status",
            "created_time": "2010-08-02T21:27:44+0000",
            "updated_time": "2010-08-02T21:27:44+0000"
        }
    ]
} 

Now, I would want to split this JSON array into different events. In this message, data from-
{
"id": "",

onwards would constitute different messages.

I have tried the following in props.conf, but all the data are shows up as single event.

[ json_split ]
TIME_FORMAT=%Y-%m-%dT%H:%M:%S+%4N
TIME_PREFIX="updated_time":\s"
SHOULD_LINEMERGE=true
BREAK_ONLY_BEFORE=(\{\s+"id")
NO_BINARY_CHECK=true

Can you please tell how to split this JSON array into different events.

0 Karma

aljohnson_splun
Splunk Employee
Splunk Employee



For your sample data, I changed

BREAK_ONLY_BEFORE="id":\s".*?",

and it worked - its not very robust however - if you ever have a field that comes after "id" in the "from" node, it will break.

0 Karma

shanksholla
Explorer

Thanks for your reply!
But with this, the leading curly braces (prior to "id") is left in previous event.
Also with this, it wouldn't be possible to parse the data in events into key value pairs using INDEXED_EXTRACTIONS or KV_MODE.
Please correct me if I'm wrong

0 Karma

aljohnson_splun
Splunk Employee
Splunk Employee

Hmm. I see what your saying. I'm guessing it would be easier to either use LINE_BREAKER and turn SHOULD_LINEMERGE to false. I'll give it another go and see if I can come up with a better solution.

Could you maybe parse the events before indexing - it might be easier 🙂 ?

0 Karma

shanksholla
Explorer

Can you please tell me how to parse before indexing. Is it using transforms.conf?

0 Karma
Career Survey
First 500 qualified respondents will receive a $20 gift card! Tell us about your professional Splunk journey.

Can’t make it to .conf25? Join us online!

Get Updates on the Splunk Community!

Leveraging Automated Threat Analysis Across the Splunk Ecosystem

Are you leveraging automation to its fullest potential in your threat detection strategy?Our upcoming Security ...

Can’t Make It to Boston? Stream .conf25 and Learn with Haya Husain

Boston may be buzzing this September with Splunk University and .conf25, but you don’t have to pack a bag to ...

Splunk Lantern’s Guide to The Most Popular .conf25 Sessions

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...