Splunk Performance Problems in Dashboard

mpepem — Thu, 09 Feb 2017 19:54:47 GMT

Hi everyone, I'm new to Splunk and I developed a Dashboard which has 19 panels, the Dashboard it's done using post-process with a Base Search and specific queries for every Panel. I'm experiencing serious performance problems with this Dashboard, the Dashboard's Panels, when selecting 4 hours interval (for example), took about 20 minutes to finish. Can anyone help me with this? maybe it's a better way to do the Base Search or the Dashboard.

I'm getting logs from a F5 equipment, specifically http logs, which are indexed by Splunk, and the Base Search and all the queries are done from this logs.
The indexed rate is about 270 KB/s.

I'm also working with tokens which are update with some drilldowns and I'm using some java scripting for some drilldowns.

I pasted the Base Search and one of the Panel configuration.

Thanks in advance,
Marcelo

<query>sourcetype=f5logs [search sourcetype=f5logs | iplocation src_ip | search Country=$country_tok$ | table src_ip] node=$nodes_tok$ http_status=$http_status_tok$ cookie=$cookie_tok$ http_user_agent=$ua_tok$ http_uri=$http_uri_tok$ src_ip=$ip_tok$ http_method=$http_method_tok$ ClassC=$classC_tok$ BlockReason=$block_reason_tok$ requestVcookie=$vcookie_tok$ http_uri_query=$uri_query_tok$ | fields _time, node, http_status, cookie, http_user_agent, requestVheader, http_uri, src_ip, http_method , ClassC, requestVcookie, http_uri_query, BlockReason</query>
<earliest>$time_tok.earliest$</earliest>
<latest>$time_tok.latest$</latest>




<panel>
  <chart>
    <title>Http status by time</title>
    <search base="baseSearch" id="subsearch1">
      <query>
        timechart sum(count) by http_status
      </query>
    </search>
    <selection>
      <set token="selection_earliest_tok">$start$</set>
      <set token="selection_latest_tok">$end$</set>
    </selection>
    <drilldown>
      <set token="http_status_tok">$click.name2$</set>
    </drilldown>
    <option name="charting.chart.stackMode">stacked</option>
    <option name="refresh.display">preview</option>
  </chart>
  <html>
          <input id="btn1" class="btnModalInfo" data-toggle="modal" data-target="#desc1" type="button" value="Detail"/>

  <!-- Modal -->
  <div class="modal fade" id="desc1" tabindex="-1" role="dialog" aria-labelledby="myModalLabel" aria-hidden="true">
      <div class="modal-dialog modal-lg" role="document">
        <div class="modal-content">
          <div class="modal-header">
            <h4 class="modal-title" id="myModalLabel">
              Http status detail view</h4>
            </div>
            <div class="modal-body">
              <div id="table1"></div>

            </div>
            <div class="modal-footer">
              <button type="button" class="btn btn-default" data-dismiss="modal">Close</button>
            </div>
          </div>
        </div>
  </div>
</html>
</panel>

Re: Splunk Performance Problems in Dashboard

ehudb — Thu, 09 Feb 2017 22:35:31 GMT

It's not recommended to use a base search without statistics involved - stats\timechart\chart
The reason is the first search causes a Splunk job to retrieve all raw data from these events, while the second one only need few fields.

I recommend to use timechart in the first base search,, if that helps sum subsearches (as Splunk calls them: post-process)
If only one post process search can use that, consider running the searches directly in each panel without that base search.

Maybe an accelerated report or a summary index could help more to achieve better performance.

Re: Splunk Performance Problems in Dashboard

mpepem — Fri, 10 Feb 2017 13:59:53 GMT

Thanks ehudb for your answer, I'll try it and keep you update.

Re: Splunk Performance Problems in Dashboard

mpepem — Fri, 10 Feb 2017 15:20:06 GMT

I have another question about what you've said because I could do the same Base Search but using "stats count by" instead of using "fields", do you think that will be better to use and will experience a better performance?

Before asking here and moving to "fields" I had used "stats" in that Base Search and the performance was more-less the same.

Thanks,
Marcelo

topic Splunk Performance Problems in Dashboard in Monitoring Splunk

Splunk Performance Problems in Dashboard

Re: Splunk Performance Problems in Dashboard

Re: Splunk Performance Problems in Dashboard

Re: Splunk Performance Problems in Dashboard