topic Re: Why is the JSON index field extraction failing with large events (> 10k bytes)? in Splunk Search

Why is the JSON index field extraction failing with large events (> 10k bytes)?

ecd — Mon, 13 Aug 2018 03:07:02 GMT

I'm using indexed field extraction to ingest JSON data over the HTTP Event Collector.

It works great. Except, once the event is > 10k bytes, the fields within the JSON are not indexed automatically. For example, if I submit a 15k event then search for it via host, I am able to find it. However, if I search for it via a field within the JSON, it does not come up.

Is it possible to configure this setting? I haven't seen anything in the documentation yet. I'm still new to this particular functionality

Thanks

Re: Why is the JSON index field extraction failing with large events (> 10k bytes)?

brian_rampley — Mon, 13 Aug 2018 07:10:15 GMT

Do the events appear complete when you search for them via "host"? Meaning, the JSON does not appear truncated in the event viewer. I would imagine that you are running up against the default TRUNCATE option for your sourcetype (in props.conf), which by default is set to 10000 bytes. I would try setting TRUNCATE for your sourcetype higher, and then coming back here if that does not work.

Re: Why is the JSON index field extraction failing with large events (> 10k bytes)?

ecd — Mon, 13 Aug 2018 17:26:27 GMT

The events do appear and are complete. We identified the issue - I'll add an answer for our fix

Re: Why is the JSON index field extraction failing with large events (> 10k bytes)?

ecd — Mon, 13 Aug 2018 17:29:15 GMT

We fixed this by explicitly setting

[json]
KV_MODE = json

It appears when unset and implicitly using KV mode, this 10k limit is hit.

Re: Why is the JSON index field extraction failing with large events (> 10k bytes)?

nm1984splunk — Mon, 11 Feb 2019 12:40:23 GMT

Hi @ecd, which version of splunk you are using ? i am assuming this stanza was created in any props.conf on splunk that is hosting HEC tokens ?

Re: Why is the JSON index field extraction failing with large events (> 10k bytes)?

vasanthi77 — Wed, 24 Jul 2019 06:29:59 GMT

Hi Ecd ,

even i m facing the same issue. can u please tell in where you have configured?(indexder, HF,SH)

Thanks in advance