Join the Conversation
- Find Answers
- :
- Splunk Administration
- :
- Monitoring Splunk
- :
- Splunk Performance Problems in Dashboard
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark Topic
- Subscribe to Topic
- Mute Topic
- Printer Friendly Page
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Splunk Performance Problems in Dashboard
Hi everyone, I'm new to Splunk and I developed a Dashboard which has 19 panels, the Dashboard it's done using post-process with a Base Search and specific queries for every Panel. I'm experiencing serious performance problems with this Dashboard, the Dashboard's Panels, when selecting 4 hours interval (for example), took about 20 minutes to finish. Can anyone help me with this? maybe it's a better way to do the Base Search or the Dashboard.
I'm getting logs from a F5 equipment, specifically http logs, which are indexed by Splunk, and the Base Search and all the queries are done from this logs.
The indexed rate is about 270 KB/s.
I'm also working with tokens which are update with some drilldowns and I'm using some java scripting for some drilldowns.
I pasted the Base Search and one of the Panel configuration.
Thanks in advance,
Marcelo
<query>sourcetype=f5logs [search sourcetype=f5logs | iplocation src_ip | search Country=$country_tok$ | table src_ip] node=$nodes_tok$ http_status=$http_status_tok$ cookie=$cookie_tok$ http_user_agent=$ua_tok$ http_uri=$http_uri_tok$ src_ip=$ip_tok$ http_method=$http_method_tok$ ClassC=$classC_tok$ BlockReason=$block_reason_tok$ requestVcookie=$vcookie_tok$ http_uri_query=$uri_query_tok$ | fields _time, node, http_status, cookie, http_user_agent, requestVheader, http_uri, src_ip, http_method , ClassC, requestVcookie, http_uri_query, BlockReason</query>
<earliest>$time_tok.earliest$</earliest>
<latest>$time_tok.latest$</latest>
<panel>
<chart>
<title>Http status by time</title>
<search base="baseSearch" id="subsearch1">
<query>
timechart sum(count) by http_status
</query>
</search>
<selection>
<set token="selection_earliest_tok">$start$</set>
<set token="selection_latest_tok">$end$</set>
</selection>
<drilldown>
<set token="http_status_tok">$click.name2$</set>
</drilldown>
<option name="charting.chart.stackMode">stacked</option>
<option name="refresh.display">preview</option>
</chart>
<html>
<input id="btn1" class="btnModalInfo" data-toggle="modal" data-target="#desc1" type="button" value="Detail"/>
<!-- Modal -->
<div class="modal fade" id="desc1" tabindex="-1" role="dialog" aria-labelledby="myModalLabel" aria-hidden="true">
<div class="modal-dialog modal-lg" role="document">
<div class="modal-content">
<div class="modal-header">
<h4 class="modal-title" id="myModalLabel">
Http status detail view</h4>
</div>
<div class="modal-body">
<div id="table1"></div>
</div>
<div class="modal-footer">
<button type="button" class="btn btn-default" data-dismiss="modal">Close</button>
</div>
</div>
</div>
</div>
</html>
</panel>
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Thanks ehudb for your answer, I'll try it and keep you update.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
It's not recommended to use a base search without statistics involved - stats\timechart\chart
The reason is the first search causes a Splunk job to retrieve all raw data from these events, while the second one only need few fields.
I recommend to use timechart in the first base search,, if that helps sum subsearches (as Splunk calls them: post-process)
If only one post process search can use that, consider running the searches directly in each panel without that base search.
Maybe an accelerated report or a summary index could help more to achieve better performance.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I have another question about what you've said because I could do the same Base Search but using "stats count by" instead of using "fields", do you think that will be better to use and will experience a better performance?
Before asking here and moving to "fields" I had used "stats" in that Base Search and the performance was more-less the same.
Thanks,
Marcelo
Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!
Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.
Unlocking Unified Insights: New Gigamon Federated Search App for Splunk
GA: New Data Management App in Splunk Platform
Announcing Modern Navigation: A New Era of Splunk User Experience
