Re: Splunk Performance Problems in Dashboard

mpepem · ‎02-09-2017

Hi everyone, I'm new to Splunk and I developed a Dashboard which has 19 panels, the Dashboard it's done using post-process with a Base Search and specific queries for every Panel. I'm experiencing serious performance problems with this Dashboard, the Dashboard's Panels, when selecting 4 hours interval (for example), took about 20 minutes to finish. Can anyone help me with this? maybe it's a better way to do the Base Search or the Dashboard.

I'm getting logs from a F5 equipment, specifically http logs, which are indexed by Splunk, and the Base Search and all the queries are done from this logs.
The indexed rate is about 270 KB/s.

I'm also working with tokens which are update with some drilldowns and I'm using some java scripting for some drilldowns.

I pasted the Base Search and one of the Panel configuration.

Thanks in advance,
Marcelo

<query>sourcetype=f5logs [search sourcetype=f5logs | iplocation src_ip | search Country=$country_tok$ | table src_ip] node=$nodes_tok$ http_status=$http_status_tok$ cookie=$cookie_tok$ http_user_agent=$ua_tok$ http_uri=$http_uri_tok$ src_ip=$ip_tok$ http_method=$http_method_tok$ ClassC=$classC_tok$ BlockReason=$block_reason_tok$ requestVcookie=$vcookie_tok$ http_uri_query=$uri_query_tok$ | fields _time, node, http_status, cookie, http_user_agent, requestVheader, http_uri, src_ip, http_method , ClassC, requestVcookie, http_uri_query, BlockReason</query>
<earliest>$time_tok.earliest$</earliest>
<latest>$time_tok.latest$</latest>




<panel>
  <chart>
    <title>Http status by time</title>
    <search base="baseSearch" id="subsearch1">
      <query>
        timechart sum(count) by http_status
      </query>
    </search>
    <selection>
      <set token="selection_earliest_tok">$start$</set>
      <set token="selection_latest_tok">$end$</set>
    </selection>
    <drilldown>
      <set token="http_status_tok">$click.name2$</set>
    </drilldown>
    <option name="charting.chart.stackMode">stacked</option>
    <option name="refresh.display">preview</option>
  </chart>
  <html>
          <input id="btn1" class="btnModalInfo" data-toggle="modal" data-target="#desc1" type="button" value="Detail"/>

  <!-- Modal -->
  <div class="modal fade" id="desc1" tabindex="-1" role="dialog" aria-labelledby="myModalLabel" aria-hidden="true">
      <div class="modal-dialog modal-lg" role="document">
        <div class="modal-content">
          <div class="modal-header">
            <h4 class="modal-title" id="myModalLabel">
              Http status detail view</h4>
            </div>
            <div class="modal-body">
              <div id="table1"></div>

            </div>
            <div class="modal-footer">
              <button type="button" class="btn btn-default" data-dismiss="modal">Close</button>
            </div>
          </div>
        </div>
  </div>
</html>
</panel>

mpepem · ‎02-10-2017

Thanks ehudb for your answer, I'll try it and keep you update.

ehudb · ‎02-09-2017

It's not recommended to use a base search without statistics involved - stats\timechart\chart
The reason is the first search causes a Splunk job to retrieve all raw data from these events, while the second one only need few fields.

I recommend to use timechart in the first base search,, if that helps sum subsearches (as Splunk calls them: post-process)
If only one post process search can use that, consider running the searches directly in each panel without that base search.

Maybe an accelerated report or a summary index could help more to achieve better performance.

mpepem · ‎02-10-2017

I have another question about what you've said because I could do the same Base Search but using "stats count by" instead of using "fields", do you think that will be better to use and will experience a better performance?

Before asking here and moving to "fields" I had used "stats" in that Base Search and the performance was more-less the same.

Thanks,
Marcelo

Splunk Performance Problems in Dashboard

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Analytics Workspace deprecation

Splunk Developer Day Recap: Building, Publishing, and Growing on the Splunk Platform

[Puzzles] Solve, Learn, Repeat: Matching cron expressions

Join the Conversation