Getting Data In
Highlighted

How to parse json which makes up part of the event

Explorer

Hello,

We have some json being logged via log4j so part of the event is json, part is not. The log4j portion has the time stamp. I can use field extractions to get just the json by itself. The users could then use xmlkv to parse the json but I'm looking for this to be done at index time so the users don't need to do this - any suggestions?

Example of logs (all lines are log4j logging json):
2017-01-04 00:00:00.981 [loglevel] methodName- {"key1":"value1","key2":"value2","key3":"value3"}
2017-01-04 00:00:00.984 [log
level] methodName- {"key1":"value1"}
2017-01-04 00:00:00.984 [log_level] methodName - {"key1":"value1","key2":"value2"}

Thanks

0 Karma
Highlighted

Re: How to parse json which makes up part of the event

SplunkTrust
SplunkTrust

You can setup automatic key-value pair extraction at search time (index time extraction is costlier, slows indexing process and requires additional space) so that uses have the fields available to them without any inline extractions. Add this to your props.conf/transforms.conf on search heads.

props.conf

[YourSourceType]
TRANSFORMS-kvextract = jsonextract

transforms.conf

[jsonextract]
REGEX = \"(?<_KEY_1>[A-z0-9]+)\":\"(?<_VAL_1>[^\"]+)\"
0 Karma
Highlighted

Re: How to parse json which makes up part of the event

Splunk Employee
Splunk Employee

dynamic key names will slow things down even more. Why not indexed_extractions = JSON instead?

0 Karma
Highlighted

Re: How to parse json which makes up part of the event

SplunkTrust
SplunkTrust

It's not pure JSON, pretty sure INDEXED_EXTRACTIONS=json would not work.

0 Karma
Highlighted

Re: How to parse json which makes up part of the event

Legend
| <Your Base Search>
| rex field=_raw "\[log_level\] methodName[-|\s]+(?<jsonData>.*)" 
| table _time jsonData _raw



| eval message="Happy Splunking!!!"


0 Karma
Highlighted

Re: How to parse json which makes up part of the event

Builder

You can use spath to extract subfields from json. Try something like:

<your search>
| rex "^\d{4}-\d\d-\d\d \d\d:\d\d:\d\d:\d\d\d \[(?P<log_level>\w+\] (?<method>\w+) (?P<my_json>.*)$"
| spath field=my_json path=my_prefix

This will create log_level, method, my_json and the hierarchy of my_prefix.* fields (in your cases you'll get my_prefix.key_1, my_prefix.key_2 and my_prefix.key_3)

0 Karma
Highlighted

Re: How to parse json which makes up part of the event

Splunk Employee
Splunk Employee

@rkeenan - Did one of the answers below help provide a solution your question? If yes, please click “Accept” below the best answer to resolve this post and upvote anything that was helpful. If no, please leave a comment with more feedback. Thanks.

0 Karma