topic Re: Best Methods to Improve Performance of Dashboard in Splunk Search

Best Methods to Improve Performance of Dashboard

katzr — Tue, 03 Oct 2017 18:51:21 GMT

I have a dashboard with ~38 panels with 2 joins per panel. I'm curious what is the most costly for Splunk performance of a dashboard- is it the large number of panels I have or is it the number of joins I have in each?

What are some common ways to improve the performance of a dashboard? Below is an example of one of my panels. I am doing some weird things with my location info because using the default value setting in my lookup table was throwing me a weird error.

 index=example date_month=August date_year=2017 (assignment_group="*") | dedup number | fillnull value="UNKNOWN" location | eval regionblank= "UNKNOWN" | eval countryblank= "UNKNOWN" | eval locationblank="UNKNOWN" | lookup CurrentSiteInfo.csv location| eval site=coalesce(location2,locationblank) | eval Region=coalesce(Region,regionblank)| eval Country=coalesce(Country,countryblank) | search ((Region="*") (Country="*") (site="*")) |stats count as Tickets by contact_type | join overwrite=false contact_type [search index=example earliest="6/01/2017:00:00:00" latest="12/31/2017:24:00:00" (assignment_group="*") | dedup number | fillnull value="UNKNOWN" location | eval regionblank= "UNKNOWN" | eval countryblank= "UNKNOWN" | eval locationblank="UNKNOWN" | lookup CurrentSiteInfo.csv location| eval site=coalesce(location2,locationblank) | eval Region=coalesce(Region,regionblank)| eval Country=coalesce(Country,countryblank) | search ((Region="*") (Country="*") (site="*"))| bucket _time span=1mon | stats count as Tickets by contact_type _time | stats avg(Tickets) as Baseline by contact_type | eval Baseline = round(Baseline,0)] | eval "Baseline Variance" = Tickets - Baseline | join overwrite=false contact_type [search index=example earliest=-3mon@mon (assignment_group="*") | dedup number | fillnull value="UNKNOWN" location | eval regionblank= "UNKNOWN" | eval countryblank= "UNKNOWN" | eval locationblank="UNKNOWN" | lookup CurrentSiteInfo.csv location| eval site=coalesce(location2,locationblank) | eval Region=coalesce(Region,regionblank)| eval Country=coalesce(Country,countryblank) | search ((Region="*") (Country="*") (site="*")) | bucket _time span=1mon | stats count as Tickets by contact_type _time | stats avg(Tickets) as Average by contact_type | eval Average = round(Average,0)] | eval "Average Variance" = Tickets - Average | table contact_type Tickets Baseline "Baseline Variance" Average "Average Variance" | addcoltotals | sort 0 Tickets

Re: Best Methods to Improve Performance of Dashboard

adonio — Tue, 03 Oct 2017 18:58:36 GMT

ask yourself if 38 panels are really necessary.
you are tying 38 cores when you are starting this dashboard...
there are plenty of other tricks like base search and more

Re: Best Methods to Improve Performance of Dashboard

katzr — Tue, 03 Oct 2017 19:01:39 GMT

This dashboard is for a customer who requested this high number of panels. Can you expand upon the "plenty of other tricks"

Re: Best Methods to Improve Performance of Dashboard

MuS — Tue, 03 Oct 2017 19:05:25 GMT

Additionally to this advice, use a base search and post process the results http://docs.splunk.com/Documentation/Splunk/latest/Viz/Savedsearches#Post-process_searches_2
As base search use a search that skips the join and use stats instead, read here http://sideviewapps.com/slides/2017_05_02_sideview_let_stats_sort_them_out.pptx to get some create advice on how to use stats or here https://answers.splunk.com/answers/129424/how-to-compare-fields-over-multiple-sourcetypes-without-join-append-or-use-of-subsearches.html

cheers, MuS

Re: Best Methods to Improve Performance of Dashboard

adonio — Tue, 03 Oct 2017 19:08:10 GMT

so really depends on the use cases and on what the panels cover.
you can create base searches, improve searches, set panels loading order look here: https://answers.splunk.com/answers/513660/how-to-set-loading-order-for-panels.html
regardless, 38 panels is plenty, try and sort out the use cases and maybe split this dashboard to 5-6 dashboards.
you can always increase the cpu on the system if this is a possibility
hope it helps

Re: Best Methods to Improve Performance of Dashboard

DalJeanis — Tue, 03 Oct 2017 21:28:29 GMT

Okay, I know we've already reviewed and simplified that search for you.

Ah, you commented on it an hour ago. We've given you that code over there.

https://answers.splunk.com/answers/578622/how-to-make-my-search-more-efficient-help-to-remov.html

Now, if you'd let us know how the different searches are different from each other, then we can discuss how to set up a base search + postprocessing in order to meet your need.

Re: Best Methods to Improve Performance of Dashboard

katzr — Wed, 04 Oct 2017 12:45:21 GMT

Hi @DalJeanis I appreciate all of your help. A lot of the searches are the same format as the one above, but are looking at it from tickets by region, problem, category, etc. instead of contact_type. But the baseline and average parameters are the same.