topic Re: Dashboard Base Search is not working for all panels in Splunk Search

Why is the Dashboard Base Search is not working for all panels?

hackentrick — Sat, 21 Nov 2020 07:18:01 GMT

I did build a Dashboard with a base search and five panels, all based on the base search. Somehow, two of five panels are not working.

Can anybody tell me why?

Base Part - this is working with 3 of 5 Panels:

<label>3rdsearch</label>
<search id="basis">
<query>index=mail-security
    | transaction keepevicted=true icid mid
    | search policy_direction="inbound"
    | eval msec_default_threat_reason =coalesce(case(spam_verdict="positive","Spam Detected",av_verdict="positive","Virus Detected",content_filter="content filter","Stopped by Content Filter",invalid_recipient="rejected by SMTP Call-Ahead","Stopped as Invalid Recipients",msec_default_reputationfilter="REJECT SG BLACKLIST","Stopped by Reputation Filtering", vof_verdict="positive","outbreak"),"Clean Messages")</query>
</search>
<fieldset autoRun="false" submitButton="true">
<input type="time" searchWhenChanged="false" token="zeit">
    <default>
        <earliest>-15m</earliest>
        <latest>now</latest>
    </default>
</input>
</fieldset>

One of the Panels which is not working:

<row>
    <panel>
    <title>Top Domain by Total Threat Messages</title>
    <chart>
        <search base="basis">
        <query>| search NOT msec_default_threat_reason="outbreak" NOT msec_default_threat_reason="Clean Messages"
            | rex field=recipient "@(?<msec_default_recipient_domain>.+\.\w+)$"    
            | rex field=sender "@(?<msec_default_sender_domain>.+\.\w+)$"
            | top limit=10 msec_default_sender_domain countfield=Messages</query>
        </search>
    <option name="charting.legend.placement">none</option><option name="charting.legend.labelStyle.overflowMode">ellipsisMiddle</option>
    <option name="charting.layout.splitSeries">0</option><option name="charting.drilldown">all</option>
    <option name="charting.chart.style">shiny</option><option name="charting.chart.stackMode">stacked</option>
    <option name="charting.chart.sliceCollapsingThreshold">0.01</option><option name="charting.chart.nullValueMode">gaps</option>
    <option name="charting.chart.bubbleSizeBy">area</option><option name="charting.chart.bubbleMinimumSize">10</option>
    <option name="charting.chart.bubbleMaximumSize">50</option><option name="charting.chart">bar</option>
    <option name="charting.axisY2.scale">inherit</option><option name="charting.axisY2.enabled">false</option>
    <option name="charting.axisY.scale">linear</option><option name="charting.axisX.scale">linear</option>
    <option name="charting.axisTitleY2.visibility">visible</option><option name="charting.axisTitleY.visibility">visible</option>
    <option name="charting.axisTitleX.visibility">collapsed</option><option name="charting.axisLabelsY.majorUnit">1</option>
    <option name="charting.axisLabelsX.majorLabelStyle.rotation">0</option><option name="charting.axisLabelsX.majorLabelStyle.overflowMode">ellipsisNone</option>
    </chart>
    </panel>
</row>

Re: Dashboard Base Search is not working for all panels

somesoni2 — Tue, 19 Jul 2016 15:49:22 GMT

Try to add a table OR fields command in the base search with all the required fields.

Re: Dashboard Base Search is not working for all panels

apilger_splunk — Tue, 29 Sep 2020 10:16:53 GMT

It seem Splunk is not passing all result fields from a base search to a post search. This could be for performance reasons. You can force the base search to pass required fields explicit to the post search by adding a fields statement.
In your example:

index=mail-security
| transaction keepevicted=true icid mid
| search policy_direction="inbound"
| eval msec_default_threat_reason =coalesce(case(spam_verdict="positive","Spam Detected",av_verdict="positive","Virus Detected",content_filter="content filter","Stopped by Content Filter",invalid_recipient="rejected by SMTP Call-Ahead","Stopped as Invalid Recipients",msec_default_reputationfilter="REJECT SG BLACKLIST","Stopped by Reputation Filtering", vof_verdict="positive","outbreak"),"Clean Messages")
| fields field1 field2 field3

Re: Dashboard Base Search is not working for all panels

hackentrick — Wed, 20 Jul 2016 07:31:30 GMT

fields are working, thank you!

Re: Dashboard Base Search is not working for all panels

Walt_Splunk — Thu, 29 Sep 2016 18:28:47 GMT

Great idea but adding field did not work for me. I have rex in my base search. Work around is to just use full search string in each panel.

Re: Dashboard Base Search is not working for all panels

somesoni2 — Thu, 29 Sep 2016 18:47:27 GMT

In the fields command, specify all the fields from base search that you're going to use/refer in the panel searches, including the fields that you're extracting via rex. I don't think a rex command would cause it to fail. May be something else is breaking it and we can look at it if you can post your search.

Re: Dashboard Base Search is not working for all panels

Walt_Splunk — Thu, 29 Sep 2016 19:32:34 GMT

Walt_Test
Each panel post processes the base search through a separate search pipeline.

  <query>source="access_log" sourcetype="access_common" | rex field=uri_path "\/(?&lt;root&gt;[^\/]+)[\/|\s](?&lt;branch&gt;[^\/]+)[\s|\/] | fields uri_path 
    root path "</query>


<input type="time" searchWhenChanged="true">
  <default>
    <earliest>-24h</earliest>
    <latest>now</latest>
  </default>
</input>


<panel>
  <title>Title for Panel 1, shows data over time by type</title>
  <chart>
    <search base="base_search">
      <query>timechart count by branch</query>
    </search>
    <option name="charting.legend.placement">right</option>
    <option name="charting.legend.labelStyle.overflowMode">ellipsisMiddle</option>
    <option name="charting.layout.splitSeries.allowIndependentYRanges">0</option>
    <option name="charting.layout.splitSeries">0</option>
    <option name="charting.drilldown">all</option>
    <option name="charting.chart.style">shiny</option>
    <option name="charting.chart.stackMode">default</option>
    <option name="charting.chart.sliceCollapsingThreshold">0.01</option>
    <option name="charting.chart.showDataLabels">none</option>
    <option name="charting.chart.nullValueMode">gaps</option>
    <option name="charting.chart.bubbleSizeBy">area</option>
    <option name="charting.chart.bubbleMinimumSize">10</option>
    <option name="charting.chart.bubbleMaximumSize">50</option>
    <option name="charting.chart">line</option>
    <option name="charting.axisY2.scale">inherit</option>
    <option name="charting.axisY2.enabled">0</option>
    <option name="charting.axisY.scale">linear</option>
    <option name="charting.axisX.scale">linear</option>
    <option name="charting.axisTitleY2.visibility">visible</option>
    <option name="charting.axisTitleY.visibility">visible</option>
    <option name="charting.axisTitleX.visibility">visible</option>
    <option name="charting.axisLabelsX.majorLabelStyle.rotation">0</option>
    <option name="charting.axisLabelsX.majorLabelStyle.overflowMode">ellipsisNone</option>
  </chart>
</panel>
<panel>
  <title>Title for Panel 2, show cumulative totals by type</title>
  <table>
    <search>
      <query>source="access_log" sourcetype="access_common" | rex field=uri_path "\/(?&lt;root&gt;[^\/]+)[\/|\s](?&lt;branch&gt;[^\/]+)[\s|\/]" | stats count by branch</query>
    </search>
    <option name="displayRowNumbers">true</option>
  </table>
</panel>

Re: Dashboard Base Search is not working for all panels

gjanders — Fri, 30 Sep 2016 00:25:38 GMT

This is also the case in Splunk 6.5, I have requested an update to the documentation to state that the post-process searches appear to run in fast mode or another optimized method of searching. And therefore things do not work as expected when using a field that was not in the base search.

The current documentation around visualisation/saved searches states "Avoid post-process searches that reference fields not named in the base search". It does not say that it will not work!

Re: Dashboard Base Search is not working for all panels

MichaelMcAleer — Fri, 15 Sep 2017 11:07:13 GMT

Thanks garethatiag! It took me aaaaages to find out what they problem was with my post-process search, no mention of field extraction in the examples and only a footnote in the documentation, once I followed your advice it fixed my problem

Re: Dashboard Base Search is not working for all panels

gjanders — Mon, 18 Sep 2017 00:52:19 GMT

Glad I could help 🙂

Re: Dashboard Base Search is not working for all panels

dd_msearles — Fri, 22 Jun 2018 08:09:05 GMT

I was having the same issues, using the fields command ensured the fields were carried forward.
As garethatiag suggested, it seems like Splunk is running in fast mode for base searches.

Re: Dashboard Base Search is not working for all panels

MichaelRubenbau — Fri, 06 Jul 2018 07:38:17 GMT

I was having the same Problem.
For the lazy people among us "| fields *" solves the problem.
Please note that this will decrease performance slightly since all fields are carried along .

Re: Dashboard Base Search is not working for all panels

ww9rivers — Fri, 20 Nov 2020 21:58:44 GMT

First - Thank for the solution!

Second, I want to add: Adding "| fields *" to the end of the base search also works for me. So you don't really need to list out all the fields by name.

Re: Dashboard Base Search is not working for all panels

yshen — Wed, 24 Feb 2021 04:57:32 GMT

It took me at least 5 hours of experiment to make sure that I'm not making a mistake.

It's really frustrated and disappointed by such mystery and arbitrary behavior without any hint of error message! A great company can do better.

I wish customers should not always be treated as a victim of hidden secrete!

So the moral of the story,

a base search must explicitly specify all the fields that will be used by the searches using the base search by command fields or table.