Splunk Search

Why is the JSON index field extraction failing with large events (> 10k bytes)?

ecd
Explorer

I'm using indexed field extraction to ingest JSON data over the HTTP Event Collector.

It works great. Except, once the event is > 10k bytes, the fields within the JSON are not indexed automatically. For example, if I submit a 15k event then search for it via host, I am able to find it. However, if I search for it via a field within the JSON, it does not come up.

Is it possible to configure this setting? I haven't seen anything in the documentation yet. I'm still new to this particular functionality

Thanks

0 Karma
1 Solution

ecd
Explorer

We fixed this by explicitly setting

[json]
KV_MODE = json

It appears when unset and implicitly using KV mode, this 10k limit is hit.

View solution in original post

0 Karma

ecd
Explorer

We fixed this by explicitly setting

[json]
KV_MODE = json

It appears when unset and implicitly using KV mode, this 10k limit is hit.

View solution in original post

0 Karma

vasanthi77
Explorer

Hi Ecd ,

even i m facing the same issue. can u please tell in where you have configured?(indexder, HF,SH)

Thanks in advance

0 Karma

nm1984splunk
New Member

Hi @ecd, which version of splunk you are using ? i am assuming this stanza was created in any props.conf on splunk that is hosting HEC tokens ?

0 Karma

brian_rampley
Path Finder

Do the events appear complete when you search for them via "host"? Meaning, the JSON does not appear truncated in the event viewer. I would imagine that you are running up against the default TRUNCATE option for your sourcetype (in props.conf), which by default is set to 10000 bytes. I would try setting TRUNCATE for your sourcetype higher, and then coming back here if that does not work.

ecd
Explorer

The events do appear and are complete. We identified the issue - I'll add an answer for our fix

0 Karma
.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!