How to optimize very slow searching JSON events?

[gn694]

I am searching a new source of json data sent to Splunk (over HEC), and it is very, very slow.

Searching over just the past 4 hours shows 726,405 events . The search took  3  1/2 minutes.  Job inspector shows the most time (almost all of it) is being spent on command.search.kv.

Does Splunk have problems searching / extracting fields from larger json events?  Is there an event length at which Splunk starts to have issues?  I looked at the length of all events from this source over a 24 hour period, and the length of a majority of them is 1,000-1,999.

Event Length	Event Count
<1000	2,452
1,000-1,999	2,043,605
2,000-2,000	2,236
3,000-3,999	590
9,000-9,999	5

The json data is properly formatted - it is valid json.  Splunk is able to extract the fields, and I also checked with an online json format validator.

[gn694 · Accepted solution]

I have solved the issue.

I adjusted the sourcetype for this HEC input (it was just using the default "httpevent" sourcetype) - thinking that some field extractions created for other data sources were the issue - and search is much faster now.

A search over the 30 minutes since I adjusted the sourcetype took only 2.5 seconds.  A search of the 30 minutes before the change took 80 seconds.

[gcusello]

Hi @gn694 ,

good for you, see next time!

Please accept one answer for the other people of Community

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated 😉

[gcusello]

Hi @gn694,

ùhave these slow performaces only using this search or generally? 

because, usually, performaces are caused by low resources (CPUs) or too many scheduled searches that use all resources or slow disks.

In other words:

• how many CPUS are you using?
• how many scheduled searches are active (you can see them using the Monitoring Console)?
• how many IOPS has your storage (remember that Splunk requests at least 800 IOPS, better 1200)?

Then, why do you save your json data in a lookup (KV) instead in a naindex?

I don't think that the problem is the event lenght, but the other things I said.

Ciao.

Giuseppe

[gn694]

The slow searches are only with this new data source.  Other searches run fine.

I do not manage the SHC, so do not have details on the system specs - but every other search of other indexed data is fine, so it is not a problem with the SHC infrastructure.

I am searching indexed events, there is no lookup involved.