Splunk Search

Why is the JSON index field extraction failing with large events (> 10k bytes)?

ecd
Explorer

I'm using indexed field extraction to ingest JSON data over the HTTP Event Collector.

It works great. Except, once the event is > 10k bytes, the fields within the JSON are not indexed automatically. For example, if I submit a 15k event then search for it via host, I am able to find it. However, if I search for it via a field within the JSON, it does not come up.

Is it possible to configure this setting? I haven't seen anything in the documentation yet. I'm still new to this particular functionality

Thanks

0 Karma
1 Solution

ecd
Explorer

We fixed this by explicitly setting

[json]
KV_MODE = json

It appears when unset and implicitly using KV mode, this 10k limit is hit.

View solution in original post

0 Karma

ecd
Explorer

We fixed this by explicitly setting

[json]
KV_MODE = json

It appears when unset and implicitly using KV mode, this 10k limit is hit.

0 Karma

vasanthi77
Explorer

Hi Ecd ,

even i m facing the same issue. can u please tell in where you have configured?(indexder, HF,SH)

Thanks in advance

0 Karma

nm1984splunk
New Member

Hi @ecd, which version of splunk you are using ? i am assuming this stanza was created in any props.conf on splunk that is hosting HEC tokens ?

0 Karma

brian_rampley
Path Finder

Do the events appear complete when you search for them via "host"? Meaning, the JSON does not appear truncated in the event viewer. I would imagine that you are running up against the default TRUNCATE option for your sourcetype (in props.conf), which by default is set to 10000 bytes. I would try setting TRUNCATE for your sourcetype higher, and then coming back here if that does not work.

ecd
Explorer

The events do appear and are complete. We identified the issue - I'll add an answer for our fix

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...