topic Re: How to parse json which makes up part of the event in Getting Data In

How to parse json which makes up part of the event

rkeenan — Tue, 29 Sep 2020 12:16:34 GMT

Hello,

We have some json being logged via log4j so part of the event is json, part is not. The log4j portion has the time stamp. I can use field extractions to get just the json by itself. The users could then use xmlkv to parse the json but I'm looking for this to be done at index time so the users don't need to do this - any suggestions?

Example of logs (all lines are log4j logging json):
2017-01-04 00:00:00.981 [log_level] methodName- {"key1":"value1","key2":"value2","key3":"value3"}
2017-01-04 00:00:00.984 [log_level] methodName- {"key1":"value1"}
2017-01-04 00:00:00.984 [log_level] methodName - {"key1":"value1","key2":"value2"}

Thanks

Re: How to parse json which makes up part of the event

somesoni2 — Thu, 05 Jan 2017 20:51:15 GMT

You can setup automatic key-value pair extraction at search time (index time extraction is costlier, slows indexing process and requires additional space) so that uses have the fields available to them without any inline extractions. Add this to your props.conf/transforms.conf on search heads.

props.conf

[YourSourceType]
TRANSFORMS-kvextract = jsonextract

transforms.conf

[jsonextract]
REGEX = \"(?<_KEY_1>[A-z0-9]+)\":\"(?<_VAL_1>[^\"]+)\"

Re: How to parse json which makes up part of the event

niketn — Thu, 05 Jan 2017 20:53:48 GMT

| <Your Base Search>
| rex field=_raw "\[log_level\] methodName[-|\s]+(?<jsonData>.*)" 
| table _time jsonData _raw

Re: How to parse json which makes up part of the event

mreynov_splunk — Thu, 05 Jan 2017 21:22:18 GMT

dynamic key names will slow things down even more. Why not indexed_extractions = JSON instead?

Re: How to parse json which makes up part of the event

somesoni2 — Thu, 05 Jan 2017 21:33:44 GMT

It's not pure JSON, pretty sure INDEXED_EXTRACTIONS=json would not work.

Re: How to parse json which makes up part of the event

arkadyz1 — Thu, 05 Jan 2017 22:12:38 GMT

You can use spath to extract subfields from json. Try something like:

<your search>
| rex "^\d{4}-\d\d-\d\d \d\d:\d\d:\d\d:\d\d\d \[(?P<log_level>\w+\] (?<method>\w+) (?P<my_json>.*)$"
| spath field=my_json path=my_prefix

This will create log_level, method, my_json and the hierarchy of my_prefix.* fields (in your cases you'll get my_prefix.key_1, my_prefix.key_2 and my_prefix.key_3)

Re: How to parse json which makes up part of the event

aaraneta_splunk — Sat, 21 Jan 2017 19:44:08 GMT

@rkeenan - Did one of the answers below help provide a solution your question? If yes, please click “Accept” below the best answer to resolve this post and upvote anything that was helpful. If no, please leave a comment with more feedback. Thanks.