Why is the Dashboard Base Search is not working for all panels?

[hackentrick]

I did build a Dashboard with a base search and five panels, all based on the base search. Somehow, two of five panels are not working.

Can anybody tell me why?

Base Part - this is working with 3 of 5 Panels:

<label>3rdsearch</label>
<search id="basis">
<query>index=mail-security
    | transaction keepevicted=true icid mid
    | search policy_direction="inbound"
    | eval msec_default_threat_reason =coalesce(case(spam_verdict="positive","Spam Detected",av_verdict="positive","Virus Detected",content_filter="content filter","Stopped by Content Filter",invalid_recipient="rejected by SMTP Call-Ahead","Stopped as Invalid Recipients",msec_default_reputationfilter="REJECT SG BLACKLIST","Stopped by Reputation Filtering", vof_verdict="positive","outbreak"),"Clean Messages")</query>
</search>
<fieldset autoRun="false" submitButton="true">
<input type="time" searchWhenChanged="false" token="zeit">
    <default>
        <earliest>-15m</earliest>
        <latest>now</latest>
    </default>
</input>
</fieldset>

One of the Panels which is not working:

<row>
    <panel>
    <title>Top Domain by Total Threat Messages</title>
    <chart>
        <search base="basis">
        <query>| search NOT msec_default_threat_reason="outbreak" NOT msec_default_threat_reason="Clean Messages"
            | rex field=recipient "@(?<msec_default_recipient_domain>.+\.\w+)$"    
            | rex field=sender "@(?<msec_default_sender_domain>.+\.\w+)$"
            | top limit=10 msec_default_sender_domain countfield=Messages</query>
        </search>
    <option name="charting.legend.placement">none</option><option name="charting.legend.labelStyle.overflowMode">ellipsisMiddle</option>
    <option name="charting.layout.splitSeries">0</option><option name="charting.drilldown">all</option>
    <option name="charting.chart.style">shiny</option><option name="charting.chart.stackMode">stacked</option>
    <option name="charting.chart.sliceCollapsingThreshold">0.01</option><option name="charting.chart.nullValueMode">gaps</option>
    <option name="charting.chart.bubbleSizeBy">area</option><option name="charting.chart.bubbleMinimumSize">10</option>
    <option name="charting.chart.bubbleMaximumSize">50</option><option name="charting.chart">bar</option>
    <option name="charting.axisY2.scale">inherit</option><option name="charting.axisY2.enabled">false</option>
    <option name="charting.axisY.scale">linear</option><option name="charting.axisX.scale">linear</option>
    <option name="charting.axisTitleY2.visibility">visible</option><option name="charting.axisTitleY.visibility">visible</option>
    <option name="charting.axisTitleX.visibility">collapsed</option><option name="charting.axisLabelsY.majorUnit">1</option>
    <option name="charting.axisLabelsX.majorLabelStyle.rotation">0</option><option name="charting.axisLabelsX.majorLabelStyle.overflowMode">ellipsisNone</option>
    </chart>
    </panel>
</row>

[ww9rivers]

First - Thank for the solution!

Second, I want to add: Adding "| fields *" to the end of the base search also works for me. So you don't really need to list out all the fields by name.

[MichaelRubenbau]

I was having the same Problem.
For the lazy people among us "| fields *" solves the problem.
Please note that this will decrease performance slightly since all fields are carried along .

[apilger_splunk · Accepted solution]

It seem Splunk is not passing all result fields from a base search to a post search. This could be for performance reasons. You can force the base search to pass required fields explicit to the post search by adding a fields statement.
In your example:

index=mail-security
| transaction keepevicted=true icid mid
| search policy_direction="inbound"
| eval msec_default_threat_reason =coalesce(case(spam_verdict="positive","Spam Detected",av_verdict="positive","Virus Detected",content_filter="content filter","Stopped by Content Filter",invalid_recipient="rejected by SMTP Call-Ahead","Stopped as Invalid Recipients",msec_default_reputationfilter="REJECT SG BLACKLIST","Stopped by Reputation Filtering", vof_verdict="positive","outbreak"),"Clean Messages")
| fields field1 field2 field3

[dd_msearles]

I was having the same issues, using the fields command ensured the fields were carried forward.
As garethatiag suggested, it seems like Splunk is running in fast mode for base searches.

[gjanders]

This is also the case in Splunk 6.5, I have requested an update to the documentation to state that the post-process searches appear to run in fast mode or another optimized method of searching. And therefore things do not work as expected when using a field that was not in the base search.

The current documentation around visualisation/saved searches states "Avoid post-process searches that reference fields not named in the base search". It does not say that it will not work!

[MichaelMcAleer]

Thanks garethatiag! It took me aaaaages to find out what they problem was with my post-process search, no mention of field extraction in the examples and only a footnote in the documentation, once I followed your advice it fixed my problem

[yshen]

It took me at least 5 hours of experiment to make sure that I'm not making a mistake.

It's really frustrated and disappointed by such mystery and arbitrary behavior without any hint of error message! A great company can do better.

I wish customers should not always be treated as a victim of hidden secrete!

So the moral of the story,

a base search must explicitly specify all the fields that will be used by the searches using the base search by command fields or table.

[gjanders]

Glad I could help 🙂

[Walt_Splunk]

Great idea but adding field did not work for me. I have rex in my base search. Work around is to just use full search string in each panel.

[somesoni2]

In the fields command, specify all the fields from base search that you're going to use/refer in the panel searches, including the fields that you're extracting via rex. I don't think a rex command would cause it to fail. May be something else is breaking it and we can look at it if you can post your search.

[Walt_Splunk]

Walt_Test
Each panel post processes the base search through a separate search pipeline.

  <query>source="access_log" sourcetype="access_common" | rex field=uri_path "\/(?&lt;root&gt;[^\/]+)[\/|\s](?&lt;branch&gt;[^\/]+)[\s|\/] | fields uri_path 
    root path "</query>


<input type="time" searchWhenChanged="true">
  <default>
    <earliest>-24h</earliest>
    <latest>now</latest>
  </default>
</input>


<panel>
  <title>Title for Panel 1, shows data over time by type</title>
  <chart>
    <search base="base_search">
      <query>timechart count by branch</query>
    </search>
    <option name="charting.legend.placement">right</option>
    <option name="charting.legend.labelStyle.overflowMode">ellipsisMiddle</option>
    <option name="charting.layout.splitSeries.allowIndependentYRanges">0</option>
    <option name="charting.layout.splitSeries">0</option>
    <option name="charting.drilldown">all</option>
    <option name="charting.chart.style">shiny</option>
    <option name="charting.chart.stackMode">default</option>
    <option name="charting.chart.sliceCollapsingThreshold">0.01</option>
    <option name="charting.chart.showDataLabels">none</option>
    <option name="charting.chart.nullValueMode">gaps</option>
    <option name="charting.chart.bubbleSizeBy">area</option>
    <option name="charting.chart.bubbleMinimumSize">10</option>
    <option name="charting.chart.bubbleMaximumSize">50</option>
    <option name="charting.chart">line</option>
    <option name="charting.axisY2.scale">inherit</option>
    <option name="charting.axisY2.enabled">0</option>
    <option name="charting.axisY.scale">linear</option>
    <option name="charting.axisX.scale">linear</option>
    <option name="charting.axisTitleY2.visibility">visible</option>
    <option name="charting.axisTitleY.visibility">visible</option>
    <option name="charting.axisTitleX.visibility">visible</option>
    <option name="charting.axisLabelsX.majorLabelStyle.rotation">0</option>
    <option name="charting.axisLabelsX.majorLabelStyle.overflowMode">ellipsisNone</option>
  </chart>
</panel>
<panel>
  <title>Title for Panel 2, show cumulative totals by type</title>
  <table>
    <search>
      <query>source="access_log" sourcetype="access_common" | rex field=uri_path "\/(?&lt;root&gt;[^\/]+)[\/|\s](?&lt;branch&gt;[^\/]+)[\s|\/]" | stats count by branch</query>
    </search>
    <option name="displayRowNumbers">true</option>
  </table>
</panel>

[somesoni2]

Try to add a table OR fields command in the base search with all the required fields.

[hackentrick]

fields are working, thank you!