Splunk Search

Why is the Dashboard Base Search is not working for all panels?

hackentrick
Engager

I did build a Dashboard with a base search and five panels, all based on the base search. Somehow, two of five panels are not working.

Can anybody tell me why?

Base Part - this is working with 3 of 5 Panels:

<label>3rdsearch</label>
<search id="basis">
<query>index=mail-security
    | transaction keepevicted=true icid mid
    | search policy_direction="inbound"
    | eval msec_default_threat_reason =coalesce(case(spam_verdict="positive","Spam Detected",av_verdict="positive","Virus Detected",content_filter="content filter","Stopped by Content Filter",invalid_recipient="rejected by SMTP Call-Ahead","Stopped as Invalid Recipients",msec_default_reputationfilter="REJECT SG BLACKLIST","Stopped by Reputation Filtering", vof_verdict="positive","outbreak"),"Clean Messages")</query>
</search>
<fieldset autoRun="false" submitButton="true">
<input type="time" searchWhenChanged="false" token="zeit">
    <default>
        <earliest>-15m</earliest>
        <latest>now</latest>
    </default>
</input>
</fieldset>

One of the Panels which is not working:

<row>
    <panel>
    <title>Top Domain by Total Threat Messages</title>
    <chart>
        <search base="basis">
        <query>| search NOT msec_default_threat_reason="outbreak" NOT msec_default_threat_reason="Clean Messages"
            | rex field=recipient "@(?<msec_default_recipient_domain>.+\.\w+)$"    
            | rex field=sender "@(?<msec_default_sender_domain>.+\.\w+)$"
            | top limit=10 msec_default_sender_domain countfield=Messages</query>
        </search>
    <option name="charting.legend.placement">none</option><option name="charting.legend.labelStyle.overflowMode">ellipsisMiddle</option>
    <option name="charting.layout.splitSeries">0</option><option name="charting.drilldown">all</option>
    <option name="charting.chart.style">shiny</option><option name="charting.chart.stackMode">stacked</option>
    <option name="charting.chart.sliceCollapsingThreshold">0.01</option><option name="charting.chart.nullValueMode">gaps</option>
    <option name="charting.chart.bubbleSizeBy">area</option><option name="charting.chart.bubbleMinimumSize">10</option>
    <option name="charting.chart.bubbleMaximumSize">50</option><option name="charting.chart">bar</option>
    <option name="charting.axisY2.scale">inherit</option><option name="charting.axisY2.enabled">false</option>
    <option name="charting.axisY.scale">linear</option><option name="charting.axisX.scale">linear</option>
    <option name="charting.axisTitleY2.visibility">visible</option><option name="charting.axisTitleY.visibility">visible</option>
    <option name="charting.axisTitleX.visibility">collapsed</option><option name="charting.axisLabelsY.majorUnit">1</option>
    <option name="charting.axisLabelsX.majorLabelStyle.rotation">0</option><option name="charting.axisLabelsX.majorLabelStyle.overflowMode">ellipsisNone</option>
    </chart>
    </panel>
</row>
Labels (1)
1 Solution

apilger_splunk
Splunk Employee
Splunk Employee

It seem Splunk is not passing all result fields from a base search to a post search. This could be for performance reasons. You can force the base search to pass required fields explicit to the post search by adding a fields statement.
In your example:

index=mail-security
| transaction keepevicted=true icid mid
| search policy_direction="inbound"
| eval msec_default_threat_reason =coalesce(case(spam_verdict="positive","Spam Detected",av_verdict="positive","Virus Detected",content_filter="content filter","Stopped by Content Filter",invalid_recipient="rejected by SMTP Call-Ahead","Stopped as Invalid Recipients",msec_default_reputationfilter="REJECT SG BLACKLIST","Stopped by Reputation Filtering", vof_verdict="positive","outbreak"),"Clean Messages")
| fields field1 field2 field3

View solution in original post

ww9rivers
Communicator

First - Thank for the solution!

Second, I want to add: Adding "| fields *" to the end of the base search also works for me. So you don't really need to list out all the fields by name.

0 Karma

MichaelRubenbau
Engager

I was having the same Problem.
For the lazy people among us "| fields *" solves the problem.
Please note that this will decrease performance slightly since all fields are carried along .

apilger_splunk
Splunk Employee
Splunk Employee

It seem Splunk is not passing all result fields from a base search to a post search. This could be for performance reasons. You can force the base search to pass required fields explicit to the post search by adding a fields statement.
In your example:

index=mail-security
| transaction keepevicted=true icid mid
| search policy_direction="inbound"
| eval msec_default_threat_reason =coalesce(case(spam_verdict="positive","Spam Detected",av_verdict="positive","Virus Detected",content_filter="content filter","Stopped by Content Filter",invalid_recipient="rejected by SMTP Call-Ahead","Stopped as Invalid Recipients",msec_default_reputationfilter="REJECT SG BLACKLIST","Stopped by Reputation Filtering", vof_verdict="positive","outbreak"),"Clean Messages")
| fields field1 field2 field3

View solution in original post

dd_msearles
Path Finder

I was having the same issues, using the fields command ensured the fields were carried forward.
As garethatiag suggested, it seems like Splunk is running in fast mode for base searches.

0 Karma

gjanders
SplunkTrust
SplunkTrust

This is also the case in Splunk 6.5, I have requested an update to the documentation to state that the post-process searches appear to run in fast mode or another optimized method of searching. And therefore things do not work as expected when using a field that was not in the base search.

The current documentation around visualisation/saved searches states "Avoid post-process searches that reference fields not named in the base search". It does not say that it will not work!

MichaelMcAleer
Path Finder

Thanks garethatiag! It took me aaaaages to find out what they problem was with my post-process search, no mention of field extraction in the examples and only a footnote in the documentation, once I followed your advice it fixed my problem

0 Karma

yshen
Path Finder

It took me at least 5 hours of experiment to make sure that I'm not making a mistake.

It's really frustrated and disappointed by such mystery and arbitrary behavior without any hint of error message! A great company can do better.

I wish customers should not always be treated as a victim of hidden secrete!

So the moral of the story,

a base search must explicitly specify all the fields that will be used by the searches using the base search by command fields or table.

0 Karma

gjanders
SplunkTrust
SplunkTrust

Glad I could help 🙂

0 Karma

Walt_Splunk
Explorer

Great idea but adding field did not work for me. I have rex in my base search. Work around is to just use full search string in each panel.

0 Karma

somesoni2
Revered Legend

In the fields command, specify all the fields from base search that you're going to use/refer in the panel searches, including the fields that you're extracting via rex. I don't think a rex command would cause it to fail. May be something else is breaking it and we can look at it if you can post your search.

0 Karma

Walt_Splunk
Explorer

Walt_Test
Each panel post processes the base search through a separate search pipeline.

  <query>source="access_log" sourcetype="access_common" | rex field=uri_path "\/(?&lt;root&gt;[^\/]+)[\/|\s](?&lt;branch&gt;[^\/]+)[\s|\/] | fields uri_path 
    root path "</query>


<input type="time" searchWhenChanged="true">
  <default>
    <earliest>-24h</earliest>
    <latest>now</latest>
  </default>
</input>


<panel>
  <title>Title for Panel 1, shows data over time by type</title>
  <chart>
    <search base="base_search">
      <query>timechart count by branch</query>
    </search>
    <option name="charting.legend.placement">right</option>
    <option name="charting.legend.labelStyle.overflowMode">ellipsisMiddle</option>
    <option name="charting.layout.splitSeries.allowIndependentYRanges">0</option>
    <option name="charting.layout.splitSeries">0</option>
    <option name="charting.drilldown">all</option>
    <option name="charting.chart.style">shiny</option>
    <option name="charting.chart.stackMode">default</option>
    <option name="charting.chart.sliceCollapsingThreshold">0.01</option>
    <option name="charting.chart.showDataLabels">none</option>
    <option name="charting.chart.nullValueMode">gaps</option>
    <option name="charting.chart.bubbleSizeBy">area</option>
    <option name="charting.chart.bubbleMinimumSize">10</option>
    <option name="charting.chart.bubbleMaximumSize">50</option>
    <option name="charting.chart">line</option>
    <option name="charting.axisY2.scale">inherit</option>
    <option name="charting.axisY2.enabled">0</option>
    <option name="charting.axisY.scale">linear</option>
    <option name="charting.axisX.scale">linear</option>
    <option name="charting.axisTitleY2.visibility">visible</option>
    <option name="charting.axisTitleY.visibility">visible</option>
    <option name="charting.axisTitleX.visibility">visible</option>
    <option name="charting.axisLabelsX.majorLabelStyle.rotation">0</option>
    <option name="charting.axisLabelsX.majorLabelStyle.overflowMode">ellipsisNone</option>
  </chart>
</panel>
<panel>
  <title>Title for Panel 2, show cumulative totals by type</title>
  <table>
    <search>
      <query>source="access_log" sourcetype="access_common" | rex field=uri_path "\/(?&lt;root&gt;[^\/]+)[\/|\s](?&lt;branch&gt;[^\/]+)[\s|\/]" | stats count by branch</query>
    </search>
    <option name="displayRowNumbers">true</option>
  </table>
</panel>
0 Karma

somesoni2
Revered Legend

Try to add a table OR fields command in the base search with all the required fields.

0 Karma

hackentrick
Engager

fields are working, thank you!

0 Karma
.conf21 CFS Extended through 5/20!

Don't miss your chance
to share your Splunk
wisdom in-person or
virtually at .conf21!

Call for Speakers has
been extended through
Thursday, 5/20!