Monitoring Splunk

Splunk search takes long to show first results

dschroeter
Explorer

I have some problems with the time the first events showing up in splunk.

When I am doing searches to all indexers, it takes about 10 to 15 seconds until the first raw events are shown up.

After this it pulls the events very fast, so it goes up to million events very quick, but the start is really slow.

When limit the search to only one indexer or a few, it starts faster but still not really acceptable.

I checked the resource usage of all the instances even the search head is ok. It is the same on different search heads.

If I search for data where I only get a few results like 20 or so, it takes ages to show up. It seems that something is waiting for more results to show up first.

Not sure how I can explain that in a better way.

I noticed that the  startup.handoff time is always high

 

It is a bit faster with data in warm, but only 1 or 2 seconds. That should be normal as warm is on SSD and cold on normal disks. 

 

Thx

Labels (2)
0 Karma

jeff
Contributor

This likely is tied to the size of your search head dispatch bundles. Try something like:

 

index=_internal sourcetype=splunkd metrics group=bundles_uploads status=success 
| stats max(replication_time_msec) AS rep_time BY host peer_name
| sort -rep_time

 

and look for how long the bundles are taking to replicate. If it's consistently longer than 10 secs (10000msec) you likely have some large lookup files or others that are contributing. You can look at commands to blacklist replication or tame the size of your lookups.

0 Karma

somesoni2
Revered Legend

Can you check the resource utilization on the indexers as well (they're the one actually fetching the data)? Also, check if your searches are following best practices (have index/sourcetype filter at base search level, not using expensive commands).

0 Karma
Get Updates on the Splunk Community!

Stay Connected: Your Guide to July and August Tech Talks, Office Hours, and Webinars!

Dive into our sizzling summer lineup for July and August Community Office Hours and Tech Talks. Scroll down to ...

Edge Processor Scaling, Energy & Manufacturing Use Cases, and More New Articles on ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Get More Out of Your Security Practice With a SIEM

Get More Out of Your Security Practice With a SIEMWednesday, July 31, 2024  |  11AM PT / 2PM ETREGISTER ...