How do I optimize the performance of my dashboard with over 30 panels using post process searches?

[bshamsian]

I have a dashboard that has over 30 panels - they all have the same basic search query so I decided to use the new search optimization of Splunk 6.2. Here is what I did - I defined a global search at top as such:

<search id="baseSearch">
    <query>
        index=main sourcetype=Auth Realm=*network.org NASId=* | eval pckStatus=if(isnull(p1_pck_Status),"Unknown", p1_pck_Status)
    </query>
    <earliest>-7d@h</earliest>
    <latest>@d</latest>
  </search>

Note that the above query usually takes 3 minutes to run by itself when run in Search & reporting directly.

Now that I have the global search defined in each panel I reference it using the following query and I perform a timechart of the resulting values.

        <search base="baseSearch">
          <query> search NASId="XYZ_897*" | timechart span=1d count by pckStatus</query>
        </search>

There are about 30 panels on the page that use the same search above just with a variation in the NASID field.

After I made these changes the Dashboard hangs and the search head it is running on becomes unresponsive. In the version before I made my modification there would be 30 search queries running concurrently and maxing out number of searches on the server until they all finish. I am trying to change the dashboard to run with one Search Query and use post processing for the rest but it is not working. What am I doing wrong? Any help or documentation is appreciated.

[bshamsian]

This is a follow up to suggestion from lguinn as to using buckets/stats and chart in order to achieve what I was looking to do.

My original (Updated) Global Search and Subsearch:

<search id="baseSearch">
    <query>
          index=main sourcetype=Auth Realm=*network.org NASId=* | eval pckStatus=if(isnull(p1_pck_Status),"Unknown", p1_pck_Status)
    </query>
    <earliest>-7d@d</earliest>
    <latest>@d</latest>
</search>

Panel queries:

<search base="baseSearch">
    <query> search NASId="XYZ_897*" | timechart span=1h count by pckStatus</query>
</search>

This produces the following chart:

Using lguinn suggestion (Updated):

<search id="baseSearch">
    <query>
                 index=main sourcetype=Auth Realm=*network.org NASId=* 
                | eval pckStatus=if(isnull(p1_pck_Status),"Unknown", p1_pck_Status)
                | bucket _time span=1h
                | stats count by _time pckStatus NASId
    </query>
    <earliest>-7d@d</earliest>
    <latest>@d</latest>
</search>

Panel queries:

<search base="baseSearch">
     <query> search NASId="XYZ_897*" | chart sum(count) as count by _time pckStatus</query>
</search>

Produces the following chart - Notice no tick marks on the X axis so the viewer would not know this is over 7 days and what day starts when:

Any help in ability to show and control X Axis tick marks so there is one per day when using Chart would be greatly appreciated.

[splunk_sv]

It's been a long time, but I am just adding a comment which can be useful if someone visits this issue.
You can use the span option with chart command as well.
To refer to all the bin options, you can visit the following link,

https://docs.splunk.com/Documentation/Splunk/7.2.6/SearchReference/Chart

[lguinn2]

Could you do this instead

 <search base="baseSearch">
      <query> search NASId="XYZ_897*" | timechart sum(count) as count by pckStatus</query>
 </search>

[lguinn2 · Accepted solution]

First, I think you should read the "Searches power dashboards and forms" section in the Dashboards and Visualizations manual. In particular, read the section on Post-process searches. There are several useful recommendations there.

I think you can change your base search to

<search id="baseSearch">
    <query>
        index=main sourcetype=Auth Realm=*network.org NASId=* 
       | eval pckStatus=if(isnull(p1_pck_Status),"Unknown", p1_pck_Status)
       | bucket _time span=1d
       | stats count by _time pckStatus NASId
    </query>
  <earliest>-7d@h</earliest>
  <latest>@d</latest>
</search>

And then the panel queries would be

<search base="baseSearch">
    <query> search NASId="XYZ_897*" | chart sum(count) as count by _time pckStatus</query>
</search>

[bshamsian]

lguinn - I looked at the Post-process searches document and I understand that I was just pass way too many results into the sub panels - your suggestion mostly works - I been working on it all morning to get it to look like timechart but ran into issue. First of all I had a type in my original post and timechart span was 1h not 1d and that is why my original chart and the one from your suggestion looked different - that was easily fixed by changing bucket span=1h in your suggestion and it worked with one caveat. The way Chart automatically decided how to place tick marks on the x axis is different the way timechart does. Chart will try to put one tick mark at every hour since span is by the hour and after it loads 3 days worth of data it will end up hiding the x axis tick marks because there is too many. Timechart automatically switches to showing one tick mark per day which is the desired behavior. I've been looking all morning to see if there is a way to get chart displayed in a panel to change the way it is placing tick marks and i've not been successful yet. So far it looks like I should be able to specify the tick mark spacing using the following chart options:

        <option name="charting.axisLabelsX.majorUnit"> P0Y0M1DT0H0M0S</option>
        <option name="charting.scaleX">1</option>

But the above options have no effect. I will past pictures to further show what is looks like in another answer since I can not upload and paste pictures in a comment.

[bshamsian]

Ok after some more playing around I remembered from past that I can just use Timechart instead of chart in the subpanels. So by changing to this:

 <search base="baseSearch">
      <query> search NASId="XYZ_897*" | timechart span=1h sum(count)  by _time pckStatus</query>
 </search>

everything works and looks like the original dashboard but much much faster.

Thank you for your help