Field extraction: Is there a limit on the number of values a JSON multivalued field can hold in Splunk 6.2.1?

[somesoni2]

Hi All,

I am ingesting a json log file. The data contains a JSON array with multiple fields. Sample format

{
  "payload": {
    "rootfield1": "1234567890",
    "rootfield2": "SDFDFDF"
  },
  "event": [
    {
      "eventfield1": "1234567890",
      "eventprimarykey": "2377",
      "timestamp": "2015-02-18T10:48:14-0500",
      "data": "sdfdfdfdf"      
    },
     {
      "eventfield1": "1234567890",
      "eventprimarykey": "2378",
      "timestamp": "2015-02-18T10:48:14-0500",
      "data": "sdfdfdfdf"      
    },
....
...
 {
      "eventfield1": "1234567890",
      "eventprimarykey": "2377",
      "timestamp": "2015-02-18T10:48:14-0500",
      "data": "sdfdfdfdf"      
    }
  ]
}

The number of elements in "event{}" array could go up to 50.

The problem that I am facing is that when I check the no of values in the multivalued fields created for the array (event{}.eventfield1, event{}.eventprimarykey etc), the total count is never 50, even though the raw data has 50 unique elements. The count varies from 33 to 39 but never matches actual count in the raw data.

Is there any limit of no of values that a JSON multivalued field can hold?

Thanks in advanced.

[woodcock · Accepted solution]

According to this answer, there is a limit if you use INDEXED EXTRACTIONS = JSON but not if you use KV_MODE = json:

https://answers.splunk.com/answers/319059/indexed-extractions-json-limiting-multivalued-fiel.html

[jonathon]

Yes, I'm not using indexed field extractions, this was just for the UI/search extracting at search time.

What worked for us was making sure that we had the following in the props.conf definition:

KV_MODE = JSON

and also

maxcols = 2000

in limits.conf on our search heads.

This combination resolved the issue for us.

[badarsebard]

Also a good one to keep in mind along with maxcols is maxchars:

maxchars = integer
* Truncate _raw to this size and then do auto KV.
* Default: 10240 characters

[dstaulcu]

Just came across this answers article for a similar problem today. Thank you for the maxcols bit!

[suarezry]

I seem to be having the same problem. Although in my case, the max number of values for a multivalued field is 10. Did you sort this out?

link to problem

[sk314]

you have questions too? 😄