Extract all key value pairs JSON

[kwarre3036]

I have the following log example and Splunk correctly pulls the first few fields (non-nested) as well as the first value pair of the nested fields.  However, after the first field, Splunk does not seem to recognize the remaining fields.

{ "sessionId": "kevin70",
"service": "RAF",
"request": { "vendorId": "Digital", "clientId: "1234567890d" },
"response":
{ "vendorId": "Digital",
"clientId": "1234567890d",
"transactionStatus": "7000",
"transactionMessage": "Success" },
"elapsedTime": "513",
"timestamp_begin": 2021-04-26T21:33:43.893Z,
"level": "info",
"message": "SUCCESS",
"timestamp": "2021-04-26T21:33:44.406Z" }

My props.conf looks like the following:

[json_v3]
BREAK_ONLY_BEFORE = ^{
LINE_BREAKER = ^{
KV_MODE=json
NO_BINARY_CHECK = true
TZ = America/Chicago
category = Structured
description = A variant of the JSON source type, with support for nonexistent timestamps
disabled = false
pulldown_type = true
BREAK_ONLY_BEFORE_DATE =

My inputs.conf looks like this:

[monitor:///home/myuser/json_test.log]
index = personalizedoffer
source = json_test.log
sourcetype = json_v3
host = myhost

The last value pair that Splunk recognized is request.vendorId.  After that, no other fields are automatically generated.  Additionally, I have attempted to use spath by piping it to my simple search which is below:

index=personalizedoffer source="json_test.log"

I want the values of pairs represented including:

request.clientId, response.vendorId, response.clientId, response.transactionStatus, response,transactionMessage, elapsedTime, timestamp_begin, level, message, timestamp

Any help is appreciated!

[richgalloway]

If the raw data is exactly as shown then I believe the problem is caused by BREAK_ONLY_BEFORE = ^{.  It forces an event break at "{ "vendorId": "Digital",".  Try MUST_BREAK_AFTER = }$ to tell Splunk to break events after the end.

[kwarre3036]

The data actually has indentions on lines 2-13.  My original post was incorrectly pasted with data in the first position of each line.  The full event is being pulled into Splunk as I would expect, but all the key pairs are not being recognized.  I have attempted to paste below what the data actually looks like.  

{  "sessionId": "kevin70",
   "service": "RAF",
   "request": { "vendorId": "Digital", "clientId: "1234567890d" },
   "response":
    { "vendorId": "Digital",
      "clientId": "1234567890d",
      "transactionStatus": "7000",
      "transactionMessage": "Success" },
   "elapsedTime": "513",
   "timestamp_begin": 2021-04-26T21:33:43.893Z,
   "level": "info",
   "message": "SUCCESS",
   "timestamp": "2021-04-26T21:33:44.406Z" }

Thanks for taking a look.

[akheraj_splunk]

It looks like it's not extracting successfully because the "clientId is missing a closing quote.

Adding in back in and using the following settings extracts the fields successfully:

BREAK_ONLY_BEFORE = ^{ \"sessionId
MUST_BREAK_AFTER = }$
TIME_PREFIX = timestamp\"\:\ \"
KV_MODE = json

[akheraj_splunk · Accepted solution]

the value for timestamp_begin is also not enclosed by quotes. So you won't get the fields: timestamp_begin, message and level. Enclosing the value in quotes fixes it.

[kwarre3036]

This is working, but I have one follow=up question.  There are now two values for timestamp.  The first is "none" and the second is "2021-04-26T21:33:44.406Z".  It looks like this is a keyword to Splunk.  I would like the "_time" field to continue to be the time the data was indexed (current_time), but I would like to create a timestamp field with only one value = "2021-04-26T21:33:44.406Z"?  Is there a way to eliminate the value "none"?

Thanks for your help?

[kwarre3036]

I think I can do this in the transforms.conf.  That is what I will try next.