Splunk Search

How to optimize very slow searching JSON events?

gn694
Communicator

I am searching a new source of json data sent to Splunk (over HEC), and it is very, very slow.

Searching over just the past 4 hours shows 726,405 events . The search took  3  1/2 minutes.  Job inspector shows the most time (almost all of it) is being spent on command.search.kv.

Does Splunk have problems searching / extracting fields from larger json events?  Is there an event length at which Splunk starts to have issues?  I looked at the length of all events from this source over a 24 hour period, and the length of a majority of them is 1,000-1,999.

Event Length Event Count
<1000 2,452
1,000-1,999 2,043,605
2,000-2,000 2,236
3,000-3,999 590
9,000-9,999 5

 

The json data is properly formatted - it is valid json.  Splunk is able to extract the fields, and I also checked with an online json format validator.

Labels (2)
Tags (3)
0 Karma
1 Solution

gn694
Communicator

I have solved the issue.

I adjusted the sourcetype for this HEC input (it was just using the default "httpevent" sourcetype) - thinking that some field extractions created for other data sources were the issue - and search is much faster now.

A search over the 30 minutes since I adjusted the sourcetype took only 2.5 seconds.  A search of the 30 minutes before the change took 80 seconds.

View solution in original post

0 Karma

gn694
Communicator

I have solved the issue.

I adjusted the sourcetype for this HEC input (it was just using the default "httpevent" sourcetype) - thinking that some field extractions created for other data sources were the issue - and search is much faster now.

A search over the 30 minutes since I adjusted the sourcetype took only 2.5 seconds.  A search of the 30 minutes before the change took 80 seconds.

0 Karma

gcusello
Legend

Hi @gn694 ,

good for you, see next time!

Please accept one answer for the other people of Community

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated 😉

0 Karma

gcusello
Legend

Hi @gn694,

ùhave these slow performaces only using this search or generally? 

because, usually, performaces are caused by low resources (CPUs) or too many scheduled searches that use all resources or slow disks.

In other words:

  • how many CPUS are you using?
  • how many scheduled searches are active (you can see them using the Monitoring Console)?
  • how many IOPS has your storage (remember that Splunk requests at least 800 IOPS, better 1200)?

Then, why do you save your json data in a lookup (KV) instead in a naindex?

I don't think that the problem is the event lenght, but the other things I said.

Ciao.

Giuseppe

0 Karma

gn694
Communicator

The slow searches are only with this new data source.  Other searches run fine.

I do not manage the SHC, so do not have details on the system specs - but every other search of other indexed data is fine, so it is not a problem with the SHC infrastructure.

I am searching indexed events, there is no lookup involved.

0 Karma
Get Updates on the Splunk Community!

Happy CX Day to our Community Superheroes!

Happy 10th Birthday CX Day!What is CX Day? It’s a global celebration recognizing innovation and success in the ...

Check out This Month’s Brand new Splunk Lantern Articles

Splunk Lantern is a customer success center providing advice from Splunk experts on valuable data insights, ...

Routing Data to Different Splunk Indexes in the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...