topic Re: Why Does JSON Logging Make Queries Slower? in Getting Data In

Why Does JSON Logging Make Queries Slower?

splunkqy — Sun, 26 May 2019 18:06:22 GMT

I was previously logging my API requests and responses as strings. So my logs would show:

Request: {"name": "Joe", "age": "4", ...}

And the entirety of {"name": "Joe", "age": "4", ...} is a string.

In order to get the information out, I had to parse the string with regexes.

I wanted to log this with Splunk fields though. So I began logging the JSON, and my logs showed:

Request:
  name: Joe
  age: 4

The problem is that my queries now take MUCH longer. I have to search for Request.name instead of using regexes to get the name. Does anyone know why it's taking so much longer?

Re: Why Does JSON Logging Make Queries Slower?

koshyk — Sun, 26 May 2019 20:59:06 GMT

We have plenty of data coming in json, and we haven't detected it not that slow. some possible checks you could do:
1. are you indexing JSON (determining it as structured format) during indextime or search time?
2. Do NOT index data into another format. Just index as pure JSON and extract required fields at searchtime
3. You don't need regexes to parse JSON if you have indexed it correctly. or alternatively you can use spath command do do during search time
4. Sometimes it may not be pure JSON? Please validate your whole event into JSON validator/parser checker

If you put the original data and your props/transforms, then we can pinpoint the correct config

Re: Why Does JSON Logging Make Queries Slower?

splunkqy — Sun, 26 May 2019 21:22:47 GMT

I did not know there was a difference in the time of indexing. I'll look into that for #1.

4 - it is pure JSON.