Splunk UI: Some JSON Logs Not Being Parsed Correctly

[kahlerb]

I recently introduced some new fields to my boilerplate logging strategy. After this introduction, Splunk's UI is not parsing the majority of my logs as json, and instead grouping several json objects together. The only addition I have made was to add client_id as a nested key under tags.

Here is an example of a log that is parsed correctly in the Splunk UI:

{
    "tags": {
        "client_id": null,
        "req_id": "12312312312asdfasdfas"
    },
    "timestamp": "2018-05-08T23:59:54Z",
    "application": "MY APPLICATION",
    "env": "development",
    "log_level": "INFO",
    "message": "Processing by HealthCheckController#index as */*"
}

Here is an example of a log that is not parsed correctly in the Splunk UI:

{
    "tags": {
        "client_id": "12312312312",
        "req_id": "79879879879879"
    },
    "timestamp": "2018-05-08T23:59:58Z",
    "application": "MY APPLICATION",
    "env": "development",
    "log_level": "INFO",
    "message": "Completed 200 OK in 21ms (Views: 8.0ms | ActiveRecord: 5.8ms)"
}

Both are valid json. It seems that anywhere tags.client_id is not null, the log is not being parsed correctly and instead being grouped with several other logs. Though even that does not hold true 100% of the time.

What might cause Splunk to fail to parse some valid json logs, but not others ?
Does the timing of the logs have anything to do with it (Some of my logs come in bursts) ?
The grouping of some of the logs, is also impacting search queries.

[niketn]

@kahlerb, based on the sample data provided spath works just fine on the second JSON data. Following is a run anywhere example for you to test.

| makeresults
| eval _raw=" {
     \"tags\": {
         \"client_id\": \"12312312312\",
         \"req_id\": \"79879879879879\"
     },
     \"timestamp\": \"2018-05-08T23:59:58Z\",
     \"application\": \"MY APPLICATION\",
     \"env\": \"development\",
     \"log_level\": \"INFO\",
     \"message\": \"Completed 200 OK in 21ms (Views: 8.0ms | ActiveRecord: 5.8ms)\"
 }"
 | spath

[MuS]

Hi @niketnilay,

your example is search time and works just fine, but the question asked is all related to parsing of the events 😉

cheers, MuS

[niketn]

@MuS, yes I was just trying to show that there is no issue with parsing of JSON. Hence issue was implied to be with data input (props.conf), as you have pointed out. I noticed you pin-pointed the issue after I posted my answer 🙂

[MuS · Accepted solution]

Hi kahlerb,

client_id suspiciously looks like an epoch timestamp and therefore Splunk thinks it is one. Configure a props.conf on the parsing Splunk instance for the sourcetype and use the TIME_PREFIX option like this :

TIME_PREFIX="timestamp": "

You might also need to adjust the MAX_TIMESTAMP_LOOKAHEAD to get that far into the event, and TIME_FORMAT to help Splunk to understand what format your time stamp has.
See the docs http://docs.splunk.com/Documentation/Splunk/latest/Admin/Propsconf#Timestamp_extraction_configuratio... for more detail.

cheers, MuS