topic Re: Dashboard is running extremely slow. in Splunk Search

Dashboard is running extremely slow.

cheriemilk — Tue, 01 Dec 2020 09:05:40 GMT

Hi team,

I have created a dashboard with 8 panels, but it is running extremely extremely slow. how to improve the performance?

Here is the XML source:

<form> <label>CAL Template Configuration</label> <fieldset submitButton="true" autoRun="true"> <input type="time" token="field1"> <label>Please select a time range</label> <default> <earliest>@d</earliest> <latest>now</latest> </default> </input> <input type="dropdown" token="DC"> <label>Please select a data center</label> <choice value="*">All</choice> <choice value="DC02">DC02</choice> <choice value="DC04">DC04</choice> <choice value="DC08">DC08</choice> <choice value="DC10">DC10</choice> <choice value="DC12">DC12</choice> <choice value="DC15">DC15</choice> <choice value="DC16">DC16</choice> <choice value="DC17">DC17</choice> <choice value="DC18">DC18</choice> <choice value="DC19">DC19</choice> <choice value="DC22">DC22</choice> <choice value="DC23">DC23</choice> <choice value="DC41">DC41</choice> <choice value="DC42">DC42</choice> <choice value="DC44">DC44</choice> <choice value="DC48">DC48</choice> <default>*</default> </input> <input type="dropdown" token="ENV"> <label>Please select a environment</label> <choice value="pc">Production</choice> <choice value="sc">Preview</choice> <default>pc</default> </input> </fieldset> <row> <panel> <title># of Data Source Usage</title> <chart> <search> <query>index=*bizx_application AND (servername=$ENV$* OR host=$ENV$*) AND SFDC=$DC$ AND sourcetype=perf_log_bizx AND ACT=SAVE_CALIBRATION_TEMPLATE AND C_DS |dedup C_CTID |eval dataSource=replace(replace(C_DS,"\[",""),"\]","") |makemv delim="," dataSource | where dataSource!=others |stats count by dataSource</query> <earliest>$earliest$</earliest> <latest>$latest$</latest> </search> <option name="charting.chart">pie</option> <option name="charting.drilldown">none</option> <option name="refresh.display">progressbar</option> <option name="charting.chart.showLabels">true</option> <option name="charting.chart.showPercent">true</option> </chart> </panel> <panel> <title># of Rating Type Usage</title> <chart> <search> <query>index=*bizx_application AND (servername=$ENV$* OR host=$ENV$*) AND SFDC=$DC$ AND sourcetype=perf_log_bizx AND ACT=SAVE_CALIBRATION_TEMPLATE AND C_RT |dedup C_CTID |eval ratingType=replace(replace(C_RT,"\[",""),"\]","") |makemv delim="," ratingType |stats count by ratingType</query> <earliest>$earliest$</earliest> <latest>$latest$</latest> </search> <option name="charting.chart">pie</option> <option name="charting.drilldown">none</option> <option name="refresh.display">progressbar</option> <option name="charting.chart.showLabels">true</option> <option name="charting.chart.showPercent">true</option> </chart> </panel> </row> <row> <panel> <title># of Decimal Rating Usage</title> <table> <search> <query>index=*bizx_application AND (servername=$ENV$* OR host=$ENV$*) AND SFDC=$DC$ AND sourcetype=perf_log_bizx AND ACT=SAVE_CALIBRATION_TEMPLATE AND C_RT |dedup C_CTID |rex max_match=0 field=C_RT "(?P<ratingEnabled>[^\[,\]]+)" | mvexpand ratingEnabled |rex max_match=0 field=C_RTD "(?P<decimal>[^\[,]+)Decimal" |eval decimaled=if(in(ratingEnabled,decimal), 1,0 ) |stats count sum(decimaled) as decimaledCount by ratingEnabled |eval ratio%=round(decimaledCount*100/count, 2) |sort - ratio%</query> <earliest>$earliest$</earliest> <latest>$latest$</latest> </search> <option name="dataOverlayMode">none</option> <option name="drilldown">none</option> <option name="refresh.display">progressbar</option> </table> </panel> <panel> <title># of Guideline Rating Usage</title> <table> <search> <query>index=*bizx_application AND (servername=$ENV$* OR host=$ENV$*) AND SFDC=$DC$ AND sourcetype=perf_log_bizx AND ACT=SAVE_CALIBRATION_TEMPLATE AND C_RT |dedup C_CTID |rex max_match=0 field=C_RT "(?P<ratingEnabled>[^\[,\]]+)" |mvexpand ratingEnabled |rex max_match=0 field=C_RTG "(?P<guideline>[^\[,]+)Guideline" |eval guided=if(in(ratingEnabled,guideline), 1,0 ) |stats count sum(guided) as guidedCount by ratingEnabled |eval ratio%=round(guidedCount*100/count, 2) | sort - ratio%</query> <earliest>$earliest$</earliest> <latest>$latest$</latest> </search> <option name="dataOverlayMode">heatmap</option> <option name="drilldown">none</option> <option name="refresh.display">progressbar</option> </table> </panel> </row> <row> <panel> <title># of Advanced Option Usage</title> <table> <search> <query>index=*bizx_application AND (servername=$ENV$* OR host=$ENV$*) AND SFDC=$DC$ AND sourcetype=perf_log_bizx AND ACT=SAVE_CALIBRATION_TEMPLATE AND C_OA |dedup C_CTID |eval advancedOptions=replace(replace(C_OA,"\[",""),"\]","") |makemv delim="," advancedOptions |stats count by advancedOptions</query> <earliest>$earliest$</earliest> <latest>$latest$</latest> </search> <option name="dataOverlayMode">heatmap</option> <option name="drilldown">none</option> <option name="refresh.display">progressbar</option> </table> </panel> <panel> <title># of enabled user fields on List View</title> <table> <search> <query>index=*bizx_application AND (servername=$ENV$* OR host=$ENV$*) AND SFDC=$DC$ AND sourcetype=perf_log_bizx AND ACT=SAVE_CALIBRATION_TEMPLATE AND C_FL |dedup C_CTID |eval userFields=replace(replace(C_FL,"\[",""),"\]","") |makemv delim="," userFields |stats count by userFields</query> <earliest>$earliest$</earliest> <latest>$latest$</latest> </search> <option name="dataOverlayMode">heatmap</option> <option name="drilldown">none</option> <option name="refresh.display">progressbar</option> </table> </panel> </row> <row> <panel> <title>Buckets Number of Bin View</title> <table> <search> <query>index=*bizx_application AND (servername=$ENV$* OR host=$ENV$*) AND SFDC=$DC$ AND sourcetype=perf_log_bizx AND ACT=SAVE_CALIBRATION_TEMPLATE AND C_BN | dedup C_CTID | rex field=C_BN max_match=0 "(?P<bin>\w+):(?P<buckets>\d+)" | eval zip=mvzip(bin, buckets) | mvexpand zip | eval split=split(zip,",") | eval bin=mvindex(split,0), buckets=mvindex(split,1) | stats count by bin, buckets</query> <earliest>$earliest$</earliest> <latest>$latest$</latest> </search> <option name="dataOverlayMode">heatmap</option> <option name="drilldown">none</option> <option name="refresh.display">progressbar</option> </table> </panel> <panel> <title>Scale Type of Matrix View</title> <table> <search> <query>index=*bizx_application AND (servername=$ENV$* OR host=$ENV$*) AND SFDC=$DC$ AND sourcetype=perf_log_bizx AND ACT=SAVE_CALIBRATION_TEMPLATE AND C_BM | dedup C_CTID | rex field=C_BM max_match=0 "(?P<matrix>\w+\*\w+):(?P<buckets>\d+\*\d+)" | eval zip=mvzip(matrix, buckets) | mvexpand zip | eval split=split(zip,",") | eval matrix=mvindex(split,0), buckets=mvindex(split,1) | stats count by matrix, buckets</query> <earliest>$earliest$</earliest> <latest>$latest$</latest> </search> <option name="dataOverlayMode">heatmap</option> <option name="drilldown">none</option> <option name="refresh.display">progressbar</option> </table> </panel> </row> </form>

Re: Dashboard is running extremely slow.

gcusello — Tue, 01 Dec 2020 09:23:08 GMT

Hi @cheriemilk,

you have to use Post process Search (https://docs.splunk.com/Documentation/Splunk/8.1.0/Viz/Savedsearches#Post-process_searches_2) to have only one search to run iin the dashboard to specialize for each panel.

Try the below modification of your dashboard:

<form> <label>CAL Template Configuration</label> <fieldset submitButton="true" autoRun="true"> <input type="time" token="field1"> <label>Please select a time range</label> <default> <earliest>@d</earliest> <latest>now</latest> </default> </input> <input type="dropdown" token="DC"> <label>Please select a data center</label> <choice value="*">All</choice> <choice value="DC02">DC02</choice> <choice value="DC04">DC04</choice> <choice value="DC08">DC08</choice> <choice value="DC10">DC10</choice> <choice value="DC12">DC12</choice> <choice value="DC15">DC15</choice> <choice value="DC16">DC16</choice> <choice value="DC17">DC17</choice> <choice value="DC18">DC18</choice> <choice value="DC19">DC19</choice> <choice value="DC22">DC22</choice> <choice value="DC23">DC23</choice> <choice value="DC41">DC41</choice> <choice value="DC42">DC42</choice> <choice value="DC44">DC44</choice> <choice value="DC48">DC48</choice> <default>*</default> </input> <input type="dropdown" token="ENV"> <label>Please select a environment</label> <choice value="pc">Production</choice> <choice value="sc">Preview</choice> <default>pc</default> </input> </fieldset> <search id="my_search"> <query> index=*bizx_application AND (servername=$ENV$* OR host=$ENV$*) AND SFDC=$DC$ AND sourcetype=perf_log_bizx AND ACT=SAVE_CALIBRATION_TEMPLATE AND C_DS | fields _time _raw C_CTID C_RT C_RTD C_RTG C_BN C_BM </query> <earliest>$earliest$</earliest> <latest>$latest$</latest> </search> <row> <panel> <title># of Data Source Usage</title> <chart> <search base="my_search"> <query> | search C_DS | dedup C_CTID | eval dataSource=replace(replace(C_DS,"\[",""),"\]","") | makemv delim="," dataSource | where dataSource!=others | stats count by dataSource </query> </search> <option name="charting.chart">pie</option> <option name="charting.drilldown">none</option> <option name="refresh.display">progressbar</option> <option name="charting.chart.showLabels">true</option> <option name="charting.chart.showPercent">true</option> </chart> </panel> <panel> <title># of Rating Type Usage</title> <chart> <search base="my_search"> <query> | search C_RT | dedup C_CTID | eval ratingType=replace(replace(C_RT,"\[",""),"\]","") | makemv delim="," ratingType | stats count by ratingType </query> </search> <option name="charting.chart">pie</option> <option name="charting.drilldown">none</option> <option name="refresh.display">progressbar</option> <option name="charting.chart.showLabels">true</option> <option name="charting.chart.showPercent">true</option> </chart> </panel> </row> <row> <panel> <title># of Decimal Rating Usage</title> <table> <search base="my_search"> <query> | search C_RT | dedup C_CTID | rex max_match=0 field=C_RT "(?P<ratingEnabled>[^\[,\]]+)" | mvexpand ratingEnabled | rex max_match=0 field=C_RTD "(?P<decimal>[^\[,]+)Decimal" | eval decimaled=if(in(ratingEnabled,decimal), 1,0 ) | stats count sum(decimaled) as decimaledCount by ratingEnabled | eval ratio%=round(decimaledCount*100/count, 2) |sort - ratio% </query> </search> <option name="dataOverlayMode">none</option> <option name="drilldown">none</option> <option name="refresh.display">progressbar</option> </table> </panel> <panel> <title># of Guideline Rating Usage</title> <table> <search base="my_search"> <query> | search C_RT | dedup C_CTID | rex max_match=0 field=C_RT "(?P<ratingEnabled>[^\[,\]]+)" | mvexpand ratingEnabled | rex max_match=0 field=C_RTG "(?P<guideline>[^\[,]+)Guideline" | eval guided=if(in(ratingEnabled,guideline), 1,0 ) | stats count sum(guided) as guidedCount by ratingEnabled | eval ratio%=round(guidedCount*100/count, 2) | sort - ratio% </query> </search> <option name="dataOverlayMode">heatmap</option> <option name="drilldown">none</option> <option name="refresh.display">progressbar</option> </table> </panel> </row> <row> <panel> <title># of Advanced Option Usage</title> <table> <search base="my_search"> <query> | search C_OA | dedup C_CTID | eval advancedOptions=replace(replace(C_OA,"\[",""),"\]","") | makemv delim="," advancedOptions | stats count by advancedOptions </query> </search> <option name="dataOverlayMode">heatmap</option> <option name="drilldown">none</option> <option name="refresh.display">progressbar</option> </table> </panel> <panel> <title># of enabled user fields on List View</title> <table> <search base="my_search"> <query> | search C_FL | dedup C_CTID | eval userFields=replace(replace(C_FL,"\[",""),"\]","") | makemv delim="," userFields | stats count by userFields </query> </search> <option name="dataOverlayMode">heatmap</option> <option name="drilldown">none</option> <option name="refresh.display">progressbar</option> </table> </panel> </row> <row> <panel> <title>Buckets Number of Bin View</title> <table> <search base="my_search"> <query> | search C_BN | dedup C_CTID | rex field=C_BN max_match=0 "(?P<bin>\w+):(?P<buckets>\d+)" | eval zip=mvzip(bin, buckets) | mvexpand zip | eval split=split(zip,",") | eval bin=mvindex(split,0), buckets=mvindex(split,1) | stats count by bin, buckets </query> </search> <option name="dataOverlayMode">heatmap</option> <option name="drilldown">none</option> <option name="refresh.display">progressbar</option> </table> </panel> <panel> <title>Scale Type of Matrix View</title> <table> <search base="my_search"> <query> | search C_BM | dedup C_CTID | rex field=C_BM max_match=0 "(?P<matrix>\w+\*\w+):(?P<buckets>\d+\*\d+)" | eval zip=mvzip(matrix, buckets) | mvexpand zip | eval split=split(zip,",") | eval matrix=mvindex(split,0), buckets=mvindex(split,1) | stats count by matrix, buckets </query> </search> <option name="dataOverlayMode">heatmap</option> <option name="drilldown">none</option> <option name="refresh.display">progressbar</option> </table> </panel> </row> </form>

If one panel doesn't run, please check the fields in the base search, you must have all the fields in the following panels in the base search.

For more infos about using of Post Process Searces, see the Splunk Dashboard Examples App (https://splunkbase.splunk.com/app/1603/).

Ciao.

Giuseppe

Re: Dashboard is running extremely slow.

gcusello — Wed, 02 Dec 2020 07:15:55 GMT

Hi @cheriemilk,

good for you, we'll see nect time!

Ciao and happy splunking.

Giuseppe

P.S.. Karma Points are appreciated ;-9

Re: Dashboard is running extremely slow.

cheriemilk — Wed, 02 Dec 2020 08:54:31 GMT

Hi @gcusello ,

Thanks for the suggestion. I tried and the performance does improve. But why the progress bar for query in each panel looks like always in "in progress" status after 1 or 2 hours, and never ended (please refer to the attached screenshot below.) The time range I select is 'Last 24h.

Is there any other option I can use to speed up the dashboard running?

Re: Dashboard is running extremely slow.

gcusello — Wed, 02 Dec 2020 08:54:19 GMT

Hi @cheriemilk,

probably you have very many events to read every time during searches.

The only way to have best performaces is to accelerate searches.

You can do this in different ways:

if the searches in panels are always the same and doesn't take new logs every time (e.g.: yesterday or last month logs) you can schedule (in the night) a Report and embed the report in your dashboard;
otherwise, if your data are continously indexed, you should schedule a search (e.g. every 5/10 minutes) that add results of the last period to a summary index, then you can use this index for your searches in the panels;
third solutions is to use a Datamodel.

you can find more infos about search accelarations at:

https://docs.splunk.com/Documentation/Splunk/8.1.0/Report/Acceleratereports

https://docs.splunk.com/Documentation/Splunk/8.1.0/Knowledge/Aboutsummaryindexing

https://docs.splunk.com/Documentation/Splunk/8.1.0/Knowledge/Acceleratedatamodels

Ciao.

Giuseppe

Re: Dashboard is running extremely slow.

cheriemilk — Fri, 04 Dec 2020 02:54:55 GMT

@gcusello , thank you for the solutions. I am investigating.

Re: Dashboard is running extremely slow.

gcusello — Fri, 04 Dec 2020 07:44:39 GMT

Hi @cheriemilk,

good for you.

Ciao and happy splunking.

Giuseppe

P.S.: Karma Points are appreciated 😉