I recently introduced some new fields to my boilerplate logging strategy. After this introduction, Splunk's UI is not parsing the majority of my logs as json, and instead grouping several json objects together. The only addition I have made was to add client_id as a nested key under tags .
Here is an example of a log that is parsed correctly in the Splunk UI:
{
"tags": {
"client_id": null,
"req_id": "12312312312asdfasdfas"
},
"timestamp": "2018-05-08T23:59:54Z",
"application": "MY APPLICATION",
"env": "development",
"log_level": "INFO",
"message": "Processing by HealthCheckController#index as */*"
}
Here is an example of a log that is not parsed correctly in the Splunk UI:
{
"tags": {
"client_id": "12312312312",
"req_id": "79879879879879"
},
"timestamp": "2018-05-08T23:59:58Z",
"application": "MY APPLICATION",
"env": "development",
"log_level": "INFO",
"message": "Completed 200 OK in 21ms (Views: 8.0ms | ActiveRecord: 5.8ms)"
}
Both are valid json. It seems that anywhere tags.client_id is not null, the log is not being parsed correctly and instead being grouped with several other logs. Though even that does not hold true 100% of the time.
What might cause Splunk to fail to parse some valid json logs, but not others ?
Does the timing of the logs have anything to do with it (Some of my logs come in bursts) ?
The grouping of some of the logs, is also impacting search queries.
... View more