I am searching a new source of json data sent to Splunk (over HEC), and it is very, very slow.
Searching over just the past 4 hours shows 726,405 events . The search took 3 1/2 minutes. Job inspector shows the most time (almost all of it) is being spent on command.search.kv. Does Splunk have problems searching / extracting fields from larger json events? Is there an event length at which Splunk starts to have issues? I looked at the length of all events from this source over a 24 hour period, and the length of a majority of them is 1,000-1,999.
Event Length
Event Count
<1000
2,452
1,000-1,999
2,043,605
2,000-2,000
2,236
3,000-3,999
590
9,000-9,999
5
The json data is properly formatted - it is valid json. Splunk is able to extract the fields, and I also checked with an online json format validator.
... View more