Splunk Search

Field extraction: Is there a limit on the number of values a JSON multivalued field can hold in Splunk 6.2.1?

somesoni2
Revered Legend

Hi All,

I am ingesting a json log file. The data contains a JSON array with multiple fields. Sample format

{
  "payload": {
    "rootfield1": "1234567890",
    "rootfield2": "SDFDFDF"
  },
  "event": [
    {
      "eventfield1": "1234567890",
      "eventprimarykey": "2377",
      "timestamp": "2015-02-18T10:48:14-0500",
      "data": "sdfdfdfdf"      
    },
     {
      "eventfield1": "1234567890",
      "eventprimarykey": "2378",
      "timestamp": "2015-02-18T10:48:14-0500",
      "data": "sdfdfdfdf"      
    },
....
...
 {
      "eventfield1": "1234567890",
      "eventprimarykey": "2377",
      "timestamp": "2015-02-18T10:48:14-0500",
      "data": "sdfdfdfdf"      
    }
  ]
}

The number of elements in "event{}" array could go up to 50.

The problem that I am facing is that when I check the no of values in the multivalued fields created for the array (event{}.eventfield1, event{}.eventprimarykey etc), the total count is never 50, even though the raw data has 50 unique elements. The count varies from 33 to 39 but never matches actual count in the raw data.

Is there any limit of no of values that a JSON multivalued field can hold?

Thanks in advanced.

1 Solution

woodcock
Esteemed Legend

According to this answer, there is a limit if you use INDEXED EXTRACTIONS = JSON but not if you use KV_MODE = json:

https://answers.splunk.com/answers/319059/indexed-extractions-json-limiting-multivalued-fiel.html

View solution in original post

woodcock
Esteemed Legend

According to this answer, there is a limit if you use INDEXED EXTRACTIONS = JSON but not if you use KV_MODE = json:

https://answers.splunk.com/answers/319059/indexed-extractions-json-limiting-multivalued-fiel.html

View solution in original post

jonathon
Path Finder

Yes, I'm not using indexed field extractions, this was just for the UI/search extracting at search time.

What worked for us was making sure that we had the following in the props.conf definition:

KV_MODE = JSON

and also

maxcols = 2000

in limits.conf on our search heads.

This combination resolved the issue for us.

badarsebard
Communicator

Also a good one to keep in mind along with maxcols is maxchars:

maxchars = integer
* Truncate _raw to this size and then do auto KV.
* Default: 10240 characters

dstaulcu
Builder

Just came across this answers article for a similar problem today. Thank you for the maxcols bit!

0 Karma

suarezry
Builder

I seem to be having the same problem. Although in my case, the max number of values for a multivalued field is 10. Did you sort this out?

link to problem

0 Karma

sk314
Builder

you have questions too? 😄

Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!