Splunk Search

Why is the JSON index field extraction failing with large events (> 10k bytes)?

ecd
Explorer

I'm using indexed field extraction to ingest JSON data over the HTTP Event Collector.

It works great. Except, once the event is > 10k bytes, the fields within the JSON are not indexed automatically. For example, if I submit a 15k event then search for it via host, I am able to find it. However, if I search for it via a field within the JSON, it does not come up.

Is it possible to configure this setting? I haven't seen anything in the documentation yet. I'm still new to this particular functionality

Thanks

0 Karma
1 Solution

ecd
Explorer

We fixed this by explicitly setting

[json]
KV_MODE = json

It appears when unset and implicitly using KV mode, this 10k limit is hit.

View solution in original post

0 Karma

ecd
Explorer

We fixed this by explicitly setting

[json]
KV_MODE = json

It appears when unset and implicitly using KV mode, this 10k limit is hit.

0 Karma

vasanthi77
Explorer

Hi Ecd ,

even i m facing the same issue. can u please tell in where you have configured?(indexder, HF,SH)

Thanks in advance

0 Karma

nm1984splunk
New Member

Hi @ecd, which version of splunk you are using ? i am assuming this stanza was created in any props.conf on splunk that is hosting HEC tokens ?

0 Karma

brian_rampley
Path Finder

Do the events appear complete when you search for them via "host"? Meaning, the JSON does not appear truncated in the event viewer. I would imagine that you are running up against the default TRUNCATE option for your sourcetype (in props.conf), which by default is set to 10000 bytes. I would try setting TRUNCATE for your sourcetype higher, and then coming back here if that does not work.

ecd
Explorer

The events do appear and are complete. We identified the issue - I'll add an answer for our fix

0 Karma
Get Updates on the Splunk Community!

Why am I not seeing the finding in Splunk Enterprise Security Analyst Queue?

(This is the first of a series of 2 blogs). Splunk Enterprise Security is a fantastic tool that offers robust ...

Index This | What are the 12 Days of Splunk-mas?

December 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

Get Inspired! We’ve Got Validation that Your Hard Work is Paying Off

We love our Splunk Community and want you to feel inspired by all your hard work! Eric Fusilero, our VP of ...