I am searching a new source of json data sent to Splunk (over HEC), and it is very, very slow.
Searching over just the past 4 hours shows 726,405 events . The search took 3 1/2 minutes. Job inspector shows the most time (almost all of it) is being spent on command.search.kv.
Does Splunk have problems searching / extracting fields from larger json events? Is there an event length at which Splunk starts to have issues? I looked at the length of all events from this source over a 24 hour period, and the length of a majority of them is 1,000-1,999.
Event Length | Event Count |
<1000 | 2,452 |
1,000-1,999 | 2,043,605 |
2,000-2,000 | 2,236 |
3,000-3,999 | 590 |
9,000-9,999 | 5 |
The json data is properly formatted - it is valid json. Splunk is able to extract the fields, and I also checked with an online json format validator.
I have solved the issue.
I adjusted the sourcetype for this HEC input (it was just using the default "httpevent" sourcetype) - thinking that some field extractions created for other data sources were the issue - and search is much faster now.
A search over the 30 minutes since I adjusted the sourcetype took only 2.5 seconds. A search of the 30 minutes before the change took 80 seconds.
I have solved the issue.
I adjusted the sourcetype for this HEC input (it was just using the default "httpevent" sourcetype) - thinking that some field extractions created for other data sources were the issue - and search is much faster now.
A search over the 30 minutes since I adjusted the sourcetype took only 2.5 seconds. A search of the 30 minutes before the change took 80 seconds.
Hi @gn694 ,
good for you, see next time!
Please accept one answer for the other people of Community
Ciao and happy splunking
Giuseppe
P.S.: Karma Points are appreciated 😉
Hi @gn694,
ùhave these slow performaces only using this search or generally?
because, usually, performaces are caused by low resources (CPUs) or too many scheduled searches that use all resources or slow disks.
In other words:
Then, why do you save your json data in a lookup (KV) instead in a naindex?
I don't think that the problem is the event lenght, but the other things I said.
Ciao.
Giuseppe
The slow searches are only with this new data source. Other searches run fine.
I do not manage the SHC, so do not have details on the system specs - but every other search of other indexed data is fine, so it is not a problem with the SHC infrastructure.
I am searching indexed events, there is no lookup involved.