Why is the JSON index field extraction failing with large events (> 10k bytes)?

[ecd]

I'm using indexed field extraction to ingest JSON data over the HTTP Event Collector.

It works great. Except, once the event is > 10k bytes, the fields within the JSON are not indexed automatically. For example, if I submit a 15k event then search for it via host, I am able to find it. However, if I search for it via a field within the JSON, it does not come up.

Is it possible to configure this setting? I haven't seen anything in the documentation yet. I'm still new to this particular functionality

Thanks

[ecd · Accepted solution]

We fixed this by explicitly setting

[json]
KV_MODE = json

It appears when unset and implicitly using KV mode, this 10k limit is hit.

[vasanthi77]

Hi Ecd ,

even i m facing the same issue. can u please tell in where you have configured?(indexder, HF,SH)

Thanks in advance

[nm1984splunk]

Hi @ecd, which version of splunk you are using ? i am assuming this stanza was created in any props.conf on splunk that is hosting HEC tokens ?

[brian_rampley]

Do the events appear complete when you search for them via "host"? Meaning, the JSON does not appear truncated in the event viewer. I would imagine that you are running up against the default TRUNCATE option for your sourcetype (in props.conf), which by default is set to 10000 bytes. I would try setting TRUNCATE for your sourcetype higher, and then coming back here if that does not work.

[ecd]

The events do appear and are complete. We identified the issue - I'll add an answer for our fix