topic Re: Searches taking long time to show results in Monitoring Splunk

Searches taking long time to show results

power12 — Tue, 12 Mar 2024 19:59:09 GMT

I have single instance splunk environment where the license is 100 gb there is another single instance using the same license .and we get data every day around 6 GB both combined . Instance A is very fast but instance B is very slow .(both have same resources)

All searches and dashboards are really slow .For instance if I run a search to do a simple stats for 24 hrs ..it takes 25 seconds when compared to the other one which takes 2 seconds .I checked the job inspection which was showing

dispatch.evaluate.search = 12.84
dispatch.fetch.rcp.phase_0 =7.78

I want to know where should I start checking on the host and what are the steps to be taken

Re: Searches taking long time to show results

bowesmana — Wed, 13 Mar 2024 00:30:38 GMT

Is the data the same data or different?

What is the search in each case.

Take a look at the job inspector and job properties

https://www.splunk.com/en_us/blog/tips-and-tricks/splunk-clara-fication-job-inspector.html

Have a look at the phase0 job property in each case and also look at the LISPY in the search.log

Re: Searches taking long time to show results

kiran_panchavat — Wed, 13 Mar 2024 05:29:55 GMT

@power12

The best place to start is by analyzing

https://docs.splunk.com/Documentation/Splunk/latest/Search/ViewsearchjobpropertieswiththeJobInspector

Use the https://docs.splunk.com/Documentation/Splunk/latest/DMC/DMCoverview to check the health of Splunk.

Also use other OS related tools to troubleshoot system performance; vmstat, iostat, top, lsof to look for any processes hogging CPU, memory or any high iowait times on your disk array.

Here is a good explanation of calculating limits:

https://answers.splunk.com/answers/270544/how-to-calculate-splunk-search-concurrency-limit-f.html

Also check out these apps:
https://splunkbase.splunk.com/app/2632/

Check the efficiency of your users searches. The following will show you the longest running searches (by user - run it for 24hrs)

index="_audit" action="search" (id=* OR search_id=*)
| eval user=if(user=="n/a",null(),user)
| stats max(total_run_time) as total_run_time first(user) as user by search_id
| stats count perc95(total_run_time) median(total_run_time) by user 
|sort - perc95(total_run_time)

Re: Searches taking long time to show results

isoutamo — Wed, 13 Mar 2024 13:44:15 GMT

Is the data stored same way on both environments? Like are there same indexes, source types, props / transforms etc. on both environment?
Are the IO resources equally on both nodes? Are there running any other stuff that splunk on those nodes?
You should setup MC on both nodes and look from it what there are happening. Start with health check part. It will tell if there are some configurations which are not based on Splunk's requirements.