topic Re: How to break events from JSON data. in Splunk Search

How to break events from JSON data.

shanksholla — Tue, 11 Aug 2015 05:41:10 GMT

I have JSON data which I have to break into events. The data looks somewhat like:

{    "data": [       {          "id": "X999_Y999",          "from": {             "name": "foo Brady", "id": "P12"         },          "message": "Looking forward to 2010!",          "actions": [             {                "name": "Comment",                "link": "http://www.facebook.com/X999/posts/Y999"            },             {                "name": "Like",                "link": "http://www.facebook.com/M999/posts/Z999"            }          ],          "type": "status",          "created_time": "2010-08-02T21:27:44+0000",          "updated_time": "2010-08-02T21:27:44+0000"      },       {          "id": "A998_Z998",          "from": {             "name": "foo Manning", "id": "P18"         },          "message": "Where's my contract?",          "actions": [             {                "name": "Comment",                "link": "http://www.facebook.com/X998/posts/Z998"            },             {                "name": "Like",                "link": "http://www.facebook.com/X998/posts/Z998"            }          ],          "type": "status",          "created_time": "2010-08-02T21:27:44+0000",          "updated_time": "2010-08-02T21:27:44+0000"      }    ] }

I need to break this message into events at "id": "A998_Z998". That is-
}, { "id":
from this JSON message. I have tried the following for the props.conf-

BREAK_ONLY_BEFORE=,\s*{\s*"id":
INDEXED_EXTRACTIONS=json
KV_MODE=none
SHOULD_LINEMERGE=true
disabled=false
pulldown_type=true

The JSON key value pairs are extracted, but I'm not able to break the JSON message into events at the required place. Can you please help me with this?

Re: How to break events from JSON data.

tskinnerivsec — Tue, 29 Sep 2020 06:54:41 GMT

Did you try escaping some of the literal characters in your regex: BREAK_ONLY_BEFORE=,s*{s*"id":

this should be:

BREAK_ONLY_BEFORE = \,\s*\{\s*\"id\"\:

actually, I would also use \s+ instead of \s*, if you know that there will be at least one space character. The BREAK_ONLY_BEFORE directive
specifies a regex pattern, what you had in your configuration example wasn't regex, it was just a string of literal characters. \s is a space character in regex, not s. This answer is assuming what is in your example is exactly what's in your props.conf config and splunk answers didn't strip out the escape characters out for you when you posted it.

Re: How to break events from JSON data.

shanksholla — Tue, 29 Sep 2020 06:56:59 GMT

Sorry! There was a typo in props.conf properties I gave. I meant-

BREAK_ONLY_BEFORE=\,\s*\{\s*"id":

I guess splunk answers stripped the escape characters. Basically I have used regex for white space.Hope BREAK_ONLY_BEFORE statement turned out right this time 🙂
Also, as suggested have used s+ instead of s*. But no luck with the event split.

Re: How to break events from JSON data.

tskinnerivsec — Thu, 13 Aug 2015 20:38:17 GMT

instead of using one or more space characters in your regex after the { character, try using a negative character class instead like so:

\,\s+\{[^\"]+\"id\"\:

Re: How to break events from JSON data.

shanksholla — Mon, 17 Aug 2015 09:06:43 GMT

Thanks for your reply. But this didn't work. Though all these regexes work fine in any regex tool, Splunk is not recognizing them. Basically, now all the data comes in one event.

Re: How to break events from JSON data.

tskinnerivsec — Mon, 17 Aug 2015 21:36:36 GMT

did you try setting KV_MODE=json?