Some events are in Splunk, but are not searchable

[duckware]

Every event in an index has field XYZ (with a non-null positive number, no exceptions), and yet this search:

index=<index> XYZ=*

only finds 99.8% of the events.  The way to find the 'missing' 0.2% of the events is by this search:

index=<index> NOT XYZ=*

Looking at the missing event's _raw, the data is there, and extracting values from _raw (spath) works -- just not via field names in Splunk Search.  This 'error' only impacts around 0.2% of the events.

Has anyone seen anything like this before?  The event is in Splunk, just not searchable.  What do I ask the administrators here to investigate?

[scelikok]

Hi @duckware,

Splunk truncates events that bigger than 10KB as default. Four your JSON evens sourcetype this setting should be increased a higher value.

[your_sourcetype]
TRUNCATE = 20000

[scelikok]

Hi @duckware,

This shows that XYZ field extraction fails on 0.2% of the events. If you can post a sample from each searchable and not-searchable events, I can try to guess the problem.

[duckware]

I may have just noticed a pattern?  The 'not found' events are on 'large' sized JSON events (>10KB) where the keys are near the end of the JSON event.  When I search on totally different keys (near the front of that specific event), the one event is found.

Hopefully this is enough to get my administrators to fix the issue, but off hand, do you know what setting needs to be changed to get Splunk to index the entire JSON event (not just up to a point)?

[duckware]

Confirmed!  key=value past 10 KB in JSON event and Splunk can't find it.