Hi everyone, I'm new to Splunk and I developed a Dashboard which has 19 panels, the Dashboard it's done using post-process with a Base Search and specific queries for every Panel. I'm experiencing serious performance problems with this Dashboard, the Dashboard's Panels, when selecting 4 hours interval (for example), took about 20 minutes to finish. Can anyone help me with this? maybe it's a better way to do the Base Search or the Dashboard.
I'm getting logs from a F5 equipment, specifically http logs, which are indexed by Splunk, and the Base Search and all the queries are done from this logs.
The indexed rate is about 270 KB/s.
I'm also working with tokens which are update with some drilldowns and I'm using some java scripting for some drilldowns.
I pasted the Base Search and one of the Panel configuration.
Thanks in advance,
<query>sourcetype=f5logs [search sourcetype=f5logs | iplocation src_ip | search Country=$country_tok$ | table src_ip] node=$nodes_tok$ http_status=$http_status_tok$ cookie=$cookie_tok$ http_user_agent=$ua_tok$ http_uri=$http_uri_tok$ src_ip=$ip_tok$ http_method=$http_method_tok$ ClassC=$classC_tok$ BlockReason=$block_reason_tok$ requestVcookie=$vcookie_tok$ http_uri_query=$uri_query_tok$ | fields _time, node, http_status, cookie, http_user_agent, requestVheader, http_uri, src_ip, http_method , ClassC, requestVcookie, http_uri_query, BlockReason</query> <earliest>$time_tok.earliest$</earliest> <latest>$time_tok.latest$</latest> <panel> <chart> <title>Http status by time</title> <search base="baseSearch" id="subsearch1"> <query> timechart sum(count) by http_status </query> </search> <selection> <set token="selection_earliest_tok">$start$</set> <set token="selection_latest_tok">$end$</set> </selection> <drilldown> <set token="http_status_tok">$click.name2$</set> </drilldown> <option name="charting.chart.stackMode">stacked</option> <option name="refresh.display">preview</option> </chart> <html> <input id="btn1" class="btnModalInfo" data-toggle="modal" data-target="#desc1" type="button" value="Detail"/> <!-- Modal --> <div class="modal fade" id="desc1" tabindex="-1" role="dialog" aria-labelledby="myModalLabel" aria-hidden="true"> <div class="modal-dialog modal-lg" role="document"> <div class="modal-content"> <div class="modal-header"> <h4 class="modal-title" id="myModalLabel"> Http status detail view</h4> </div> <div class="modal-body"> <div id="table1"></div> </div> <div class="modal-footer"> <button type="button" class="btn btn-default" data-dismiss="modal">Close</button> </div> </div> </div> </div> </html> </panel>
It's not recommended to use a base search without statistics involved - stats\timechart\chart
The reason is the first search causes a Splunk job to retrieve all raw data from these events, while the second one only need few fields.
I recommend to use timechart in the first base search,, if that helps sum subsearches (as Splunk calls them: post-process)
If only one post process search can use that, consider running the searches directly in each panel without that base search.
Maybe an accelerated report or a summary index could help more to achieve better performance.
I have another question about what you've said because I could do the same Base Search but using "stats count by" instead of using "fields", do you think that will be better to use and will experience a better performance?
Before asking here and moving to "fields" I had used "stats" in that Base Search and the performance was more-less the same.