Splunk Search

Some events are in Splunk, but are not searchable

duckware
Explorer

Every event in an index has field XYZ (with a non-null positive number, no exceptions), and yet this search:

index=<index> XYZ=*

only finds 99.8% of the events.  The way to find the 'missing' 0.2% of the events is by this search:

index=<index> NOT XYZ=*

Looking at the missing event's _raw, the data is there, and extracting values from _raw (spath) works -- just not via field names in Splunk Search.  This 'error' only impacts around 0.2% of the events.

Has anyone seen anything like this before?  The event is in Splunk, just not searchable.  What do I ask the administrators here to investigate?

 

Labels (1)
0 Karma

scelikok
SplunkTrust
SplunkTrust

Hi @duckware,

Splunk truncates events that bigger than 10KB as default. Four your JSON evens sourcetype this setting should be increased a higher value.

[your_sourcetype]
TRUNCATE = 20000

 

If this reply helps you an upvote and "Accept as Solution" is appreciated.

scelikok
SplunkTrust
SplunkTrust

Hi @duckware,

This shows that XYZ field extraction fails on 0.2% of the events. If you can post a sample from each searchable and not-searchable events, I can try to guess the problem.

If this reply helps you an upvote and "Accept as Solution" is appreciated.
0 Karma

duckware
Explorer

I may have just noticed a pattern?  The 'not found' events are on 'large' sized JSON events (>10KB) where the keys are near the end of the JSON event.  When I search on totally different keys (near the front of that specific event), the one event is found.

Hopefully this is enough to get my administrators to fix the issue, but off hand, do you know what setting needs to be changed to get Splunk to index the entire JSON event (not just up to a point)?

0 Karma

duckware
Explorer

Confirmed!  key=value past 10 KB in JSON event and Splunk can't find it.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...