Splunk Search

Extract all key value pairs JSON

kwarre3036
Explorer

I have the following log example and Splunk correctly pulls the first few fields (non-nested) as well as the first value pair of the nested fields.  However, after the first field, Splunk does not seem to recognize the remaining fields.

{ "sessionId": "kevin70",
"service": "RAF",
"request": { "vendorId": "Digital", "clientId: "1234567890d" },
"response":
{ "vendorId": "Digital",
"clientId": "1234567890d",
"transactionStatus": "7000",
"transactionMessage": "Success" },
"elapsedTime": "513",
"timestamp_begin": 2021-04-26T21:33:43.893Z,
"level": "info",
"message": "SUCCESS",
"timestamp": "2021-04-26T21:33:44.406Z" }

My props.conf looks like the following:

[json_v3]
BREAK_ONLY_BEFORE = ^{
LINE_BREAKER = ^{
KV_MODE=json
NO_BINARY_CHECK = true
TZ = America/Chicago
category = Structured
description = A variant of the JSON source type, with support for nonexistent timestamps
disabled = false
pulldown_type = true
BREAK_ONLY_BEFORE_DATE =

My inputs.conf looks like this:

[monitor:///home/myuser/json_test.log]
index = personalizedoffer
source = json_test.log
sourcetype = json_v3
host = myhost

The last value pair that Splunk recognized is request.vendorId.  After that, no other fields are automatically generated.  Additionally, I have attempted to use spath by piping it to my simple search which is below:

index=personalizedoffer source="json_test.log"

I want the values of pairs represented including:

request.clientId, response.vendorId, response.clientId, response.transactionStatus, response,transactionMessage, elapsedTime, timestamp_begin, level, message, timestamp

Any help is appreciated!

 

Labels (1)
0 Karma
1 Solution

akheraj_splunk
Splunk Employee
Splunk Employee

the value for timestamp_begin is also not enclosed by quotes. So you won't get the fields: timestamp_begin, message and level. Enclosing the value in quotes fixes it.

View solution in original post

0 Karma

richgalloway
SplunkTrust
SplunkTrust

If the raw data is exactly as shown then I believe the problem is caused by BREAK_ONLY_BEFORE = ^{.  It forces an event break at "{ "vendorId": "Digital",".  Try MUST_BREAK_AFTER = }$ to tell Splunk to break events after the end.

---
If this reply helps you, Karma would be appreciated.
0 Karma

kwarre3036
Explorer

The data actually has indentions on lines 2-13.  My original post was incorrectly pasted with data in the first position of each line.  The full event is being pulled into Splunk as I would expect, but all the key pairs are not being recognized.  I have attempted to paste below what the data actually looks like.  

{  "sessionId": "kevin70",
   "service": "RAF",
   "request": { "vendorId": "Digital", "clientId: "1234567890d" },
   "response":
    { "vendorId": "Digital",
      "clientId": "1234567890d",
      "transactionStatus": "7000",
      "transactionMessage": "Success" },
   "elapsedTime": "513",
   "timestamp_begin": 2021-04-26T21:33:43.893Z,
   "level": "info",
   "message": "SUCCESS",
   "timestamp": "2021-04-26T21:33:44.406Z" }

Thanks for taking a look.

0 Karma

akheraj_splunk
Splunk Employee
Splunk Employee

It looks like it's not extracting successfully because the "clientId is missing a closing quote.

Adding in back in and using the following settings extracts the fields successfully:

 

BREAK_ONLY_BEFORE = ^{ \"sessionId
MUST_BREAK_AFTER = }$
TIME_PREFIX = timestamp\"\:\ \"
KV_MODE = json

 

akheraj_splunk
Splunk Employee
Splunk Employee

the value for timestamp_begin is also not enclosed by quotes. So you won't get the fields: timestamp_begin, message and level. Enclosing the value in quotes fixes it.

0 Karma

kwarre3036
Explorer

This is working, but I have one follow=up question.  There are now two values for timestamp.  The first is "none" and the second is "2021-04-26T21:33:44.406Z".  It looks like this is a keyword to Splunk.  I would like the "_time" field to continue to be the time the data was indexed (current_time), but I would like to create a timestamp field with only one value = "2021-04-26T21:33:44.406Z"?  Is there a way to eliminate the value "none"?

Thanks for your help?

0 Karma

kwarre3036
Explorer

I think I can do this in the transforms.conf.  That is what I will try next.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...