topic Re: Extract all key value pairs JSON in Splunk Search

Extract all key value pairs JSON

kwarre3036 — Tue, 27 Apr 2021 20:22:40 GMT

I have the following log example and Splunk correctly pulls the first few fields (non-nested) as well as the first value pair of the nested fields. However, after the first field, Splunk does not seem to recognize the remaining fields.

{ "sessionId": "kevin70",
"service": "RAF",
"request": { "vendorId": "Digital", "clientId: "1234567890d" },
"response":
{ "vendorId": "Digital",
"clientId": "1234567890d",
"transactionStatus": "7000",
"transactionMessage": "Success" },
"elapsedTime": "513",
"timestamp_begin": 2021-04-26T21:33:43.893Z,
"level": "info",
"message": "SUCCESS",
"timestamp": "2021-04-26T21:33:44.406Z" }

My props.conf looks like the following:

[json_v3]
BREAK_ONLY_BEFORE = ^{
LINE_BREAKER = ^{
KV_MODE=json
NO_BINARY_CHECK = true
TZ = America/Chicago
category = Structured
description = A variant of the JSON source type, with support for nonexistent timestamps
disabled = false
pulldown_type = true
BREAK_ONLY_BEFORE_DATE =

My inputs.conf looks like this:

[monitor:///home/myuser/json_test.log]
index = personalizedoffer
source = json_test.log
sourcetype = json_v3
host = myhost

The last value pair that Splunk recognized is request.vendorId. After that, no other fields are automatically generated. Additionally, I have attempted to use spath by piping it to my simple search which is below:

index=personalizedoffer source="json_test.log"

I want the values of pairs represented including:

request.clientId, response.vendorId, response.clientId, response.transactionStatus, response,transactionMessage, elapsedTime, timestamp_begin, level, message, timestamp

Any help is appreciated!

Re: Extract all key value pairs JSON

richgalloway — Tue, 27 Apr 2021 23:49:37 GMT

If the raw data is exactly as shown then I believe the problem is caused by BREAK_ONLY_BEFORE = ^{. It forces an event break at "{ "vendorId": "Digital",". Try MUST_BREAK_AFTER = }$ to tell Splunk to break events after the end.

Re: Extract all key value pairs JSON

kwarre3036 — Wed, 28 Apr 2021 12:48:22 GMT

The data actually has indentions on lines 2-13. My original post was incorrectly pasted with data in the first position of each line. The full event is being pulled into Splunk as I would expect, but all the key pairs are not being recognized. I have attempted to paste below what the data actually looks like.

Thanks for taking a look.

Re: Extract all key value pairs JSON

akheraj_splunk — Wed, 28 Apr 2021 16:32:58 GMT

It looks like it's not extracting successfully because the "clientId is missing a closing quote.

Adding in back in and using the following settings extracts the fields successfully:

BREAK_ONLY_BEFORE = ^{ \"sessionId MUST_BREAK_AFTER = }$ TIME_PREFIX = timestamp\"\:\ \" KV_MODE = json

Re: Extract all key value pairs JSON

akheraj_splunk — Wed, 28 Apr 2021 17:21:10 GMT

the value for timestamp_begin is also not enclosed by quotes. So you won't get the fields: timestamp_begin, message and level. Enclosing the value in quotes fixes it.

Re: Extract all key value pairs JSON

kwarre3036 — Wed, 28 Apr 2021 18:10:24 GMT

This is working, but I have one follow=up question. There are now two values for timestamp. The first is "none" and the second is "2021-04-26T21:33:44.406Z". It looks like this is a keyword to Splunk. I would like the "_time" field to continue to be the time the data was indexed (current_time), but I would like to create a timestamp field with only one value = "2021-04-26T21:33:44.406Z"? Is there a way to eliminate the value "none"?

Thanks for your help?

Re: Extract all key value pairs JSON

kwarre3036 — Wed, 28 Apr 2021 18:12:29 GMT

I think I can do this in the transforms.conf. That is what I will try next.