Splunk Search

Extract all key value pairs JSON

kwarre3036
Explorer

I have the following log example and Splunk correctly pulls the first few fields (non-nested) as well as the first value pair of the nested fields.  However, after the first field, Splunk does not seem to recognize the remaining fields.

{ "sessionId": "kevin70",
"service": "RAF",
"request": { "vendorId": "Digital", "clientId: "1234567890d" },
"response":
{ "vendorId": "Digital",
"clientId": "1234567890d",
"transactionStatus": "7000",
"transactionMessage": "Success" },
"elapsedTime": "513",
"timestamp_begin": 2021-04-26T21:33:43.893Z,
"level": "info",
"message": "SUCCESS",
"timestamp": "2021-04-26T21:33:44.406Z" }

My props.conf looks like the following:

[json_v3]
BREAK_ONLY_BEFORE = ^{
LINE_BREAKER = ^{
KV_MODE=json
NO_BINARY_CHECK = true
TZ = America/Chicago
category = Structured
description = A variant of the JSON source type, with support for nonexistent timestamps
disabled = false
pulldown_type = true
BREAK_ONLY_BEFORE_DATE =

My inputs.conf looks like this:

[monitor:///home/myuser/json_test.log]
index = personalizedoffer
source = json_test.log
sourcetype = json_v3
host = myhost

The last value pair that Splunk recognized is request.vendorId.  After that, no other fields are automatically generated.  Additionally, I have attempted to use spath by piping it to my simple search which is below:

index=personalizedoffer source="json_test.log"

I want the values of pairs represented including:

request.clientId, response.vendorId, response.clientId, response.transactionStatus, response,transactionMessage, elapsedTime, timestamp_begin, level, message, timestamp

Any help is appreciated!

 

Labels (1)
0 Karma
1 Solution

akheraj_splunk
Splunk Employee
Splunk Employee

the value for timestamp_begin is also not enclosed by quotes. So you won't get the fields: timestamp_begin, message and level. Enclosing the value in quotes fixes it.

View solution in original post

0 Karma

richgalloway
SplunkTrust
SplunkTrust

If the raw data is exactly as shown then I believe the problem is caused by BREAK_ONLY_BEFORE = ^{.  It forces an event break at "{ "vendorId": "Digital",".  Try MUST_BREAK_AFTER = }$ to tell Splunk to break events after the end.

---
If this reply helps you, Karma would be appreciated.
0 Karma

kwarre3036
Explorer

The data actually has indentions on lines 2-13.  My original post was incorrectly pasted with data in the first position of each line.  The full event is being pulled into Splunk as I would expect, but all the key pairs are not being recognized.  I have attempted to paste below what the data actually looks like.  

{  "sessionId": "kevin70",
   "service": "RAF",
   "request": { "vendorId": "Digital", "clientId: "1234567890d" },
   "response":
    { "vendorId": "Digital",
      "clientId": "1234567890d",
      "transactionStatus": "7000",
      "transactionMessage": "Success" },
   "elapsedTime": "513",
   "timestamp_begin": 2021-04-26T21:33:43.893Z,
   "level": "info",
   "message": "SUCCESS",
   "timestamp": "2021-04-26T21:33:44.406Z" }

Thanks for taking a look.

0 Karma

akheraj_splunk
Splunk Employee
Splunk Employee

It looks like it's not extracting successfully because the "clientId is missing a closing quote.

Adding in back in and using the following settings extracts the fields successfully:

 

BREAK_ONLY_BEFORE = ^{ \"sessionId
MUST_BREAK_AFTER = }$
TIME_PREFIX = timestamp\"\:\ \"
KV_MODE = json

 

akheraj_splunk
Splunk Employee
Splunk Employee

the value for timestamp_begin is also not enclosed by quotes. So you won't get the fields: timestamp_begin, message and level. Enclosing the value in quotes fixes it.

0 Karma

kwarre3036
Explorer

This is working, but I have one follow=up question.  There are now two values for timestamp.  The first is "none" and the second is "2021-04-26T21:33:44.406Z".  It looks like this is a keyword to Splunk.  I would like the "_time" field to continue to be the time the data was indexed (current_time), but I would like to create a timestamp field with only one value = "2021-04-26T21:33:44.406Z"?  Is there a way to eliminate the value "none"?

Thanks for your help?

0 Karma

kwarre3036
Explorer

I think I can do this in the transforms.conf.  That is what I will try next.

0 Karma
Get Updates on the Splunk Community!

Monitoring MariaDB and MySQL

In a previous post, we explored monitoring PostgreSQL and general best practices around which metrics to ...

Financial Services Industry Use Cases, ITSI Best Practices, and More New Articles ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Splunk Federated Analytics for Amazon Security Lake

Thursday, November 21, 2024  |  11AM PT / 2PM ET Register Now Join our session to see the technical ...