Dashboards & Visualizations

Issue with the search base

anjibabu
New Member

hi Friends,

below are my queries,

  1. index=perfmon source="Perfmon:LogicalDisk" counter="% Free Space" | search host = DMOPWMD1PDDB0* | eval FreeSpace =100-( Value ) | stats min(FreeSpace) as hostavg by host,instance | table host,instance,hostavg | chart min(hostavg) by host,instance

  2. index=perfmon sourcetype="Perfmon:Memory" counter="% Committed Bytes In Use" | search host = DMOPWMD1PDDB0* | timechart perc90(Value) by host limit=0 span=1m

i created the below search id's for the search and created the panels, these are working fine in search, but not working in dashboard\panels, the panels are showing "No Results", could you please advise.

search id's:

index=perfmon source="Perfmon:LogicalDisk" counter="% Free Space"
$TimeRangePkr.earliest$
$TimeRangePkr.latest$
5m
delay

true



<query>index=perfmon sourcetype="Perfmon:Memory" counter="% Committed Bytes In Use"</query>
<earliest>$TimeRangePkr.earliest$</earliest>
<latest>$TimeRangePkr.latest$</latest>
<refresh>5m</refresh>
<refreshType>delay</refreshType>
<progress>
  <set token="show_html">true</set>
</progress>
<done>
  <unset token="show_html"></unset>
</done>

Panels:

<panel>
  <chart>
    <title>DISK%</title>
    <search base="Disk1">
      <query>| search host = DMOPWMD1PDDB0* | eval FreeSpace =100-( Value )  | stats min(FreeSpace) as hostavg by host,instance | table host,instance,hostavg | chart min(hostavg) by host,instance</query>
      </search>
    <option name="charting.axisLabelsX.majorLabelStyle.overflowMode">ellipsisNone</option>
    <option name="charting.axisLabelsX.majorLabelStyle.rotation">0</option>
    <option name="charting.axisTitleX.text">TIME</option>
    <option name="charting.axisTitleX.visibility">visible</option>
    <option name="charting.axisTitleY.text">HOST</option>
    <option name="charting.axisTitleY.visibility">visible</option>
    <option name="charting.axisTitleY2.visibility">visible</option>
    <option name="charting.axisX.scale">linear</option>
    <option name="charting.axisY.scale">linear</option>
    <option name="charting.axisY2.enabled">0</option>
    <option name="charting.axisY2.scale">inherit</option>
    <option name="charting.chart">column</option>
    <option name="charting.chart.bubbleMaximumSize">50</option>
    <option name="charting.chart.bubbleMinimumSize">10</option>
    <option name="charting.chart.bubbleSizeBy">area</option>
    <option name="charting.chart.nullValueMode">gaps</option>
    <option name="charting.chart.sliceCollapsingThreshold">0.01</option>
    <option name="charting.chart.stackMode">default</option>
    <option name="charting.chart.style">shiny</option>
    <option name="charting.drilldown">all</option>
    <option name="charting.layout.splitSeries">0</option>
    <option name="charting.legend.labelStyle.overflowMode">ellipsisMiddle</option>
    <option name="charting.legend.placement">right</option>
    <option name="refresh.display">progressbar</option>
  </chart>
</panel>

<panel>
  <chart>
    <title>MEMORY%</title>
    <search base="Mem">
      <query>| search host = DMOPWMD1PDDB0* | timechart perc90(Value) by host limit=0 span=1m</query>
      </search>
    <option name="charting.axisLabelsX.majorLabelStyle.overflowMode">ellipsisNone</option>
    <option name="charting.axisLabelsX.majorLabelStyle.rotation">0</option>
    <option name="charting.axisTitleX.text">TIME</option>
    <option name="charting.axisTitleX.visibility">visible</option>
    <option name="charting.axisTitleY.text">HOST</option>
    <option name="charting.axisTitleY.visibility">visible</option>
    <option name="charting.axisTitleY2.visibility">visible</option>
    <option name="charting.axisX.scale">linear</option>
    <option name="charting.axisY.scale">linear</option>
    <option name="charting.axisY2.enabled">0</option>
    <option name="charting.axisY2.scale">inherit</option>
    <option name="charting.chart">line</option>
    <option name="charting.chart.bubbleMaximumSize">50</option>
    <option name="charting.chart.bubbleMinimumSize">10</option>
    <option name="charting.chart.bubbleSizeBy">area</option>
    <option name="charting.chart.nullValueMode">gaps</option>
    <option name="charting.chart.sliceCollapsingThreshold">0.01</option>
    <option name="charting.chart.stackMode">default</option>
    <option name="charting.chart.style">shiny</option>
    <option name="charting.drilldown">all</option>
    <option name="charting.layout.splitSeries">0</option>
    <option name="charting.legend.labelStyle.overflowMode">ellipsisMiddle</option>
    <option name="charting.legend.placement">right</option>
    <option name="refresh.display">progressbar</option>
  </chart>
</panel>
0 Karma

DalJeanis
SplunkTrust
SplunkTrust

The answer is thoroughly covered by @wyfwa4. You should accept that one, and upvote this one.

Here's the tl;dr -

A base search always runs in fast mode, which optimizes out everything it doesn't need. It cannot see the post process searches, and pays no attention to them. If you don't tell the base search which fields to keep, the base search will throw them all away.

Add a table or fields command at the end of the base search to keep each field you want to use in any of the post-process searches.

 | fields host instance counter Value
0 Karma

wyfwa4
Communicator

From my interpretation of your base search - the issue is that the base search is not transforming. This is a requirement for post-processed searches

https://docs.splunk.com/Documentation/Splunk/8.0.3/Viz/Savedsearches#Post-process_searches_2

If you take a simple search with stats/chart command and then run it in the standard search window, you will get the results you want. However if you split this in a form/dashboard and only have the initial search in the base search, you will not get any results from your post processing. You will need to add a stats command or similar to the base search to generate a table of results before this will work.

If I take one of your examples, the full query is as follows

index=perfmon source="Perfmon:LogicalDisk" counter="% Free Space" | search host = DMOPWMD1PDDB0* | eval FreeSpace =100-( Value ) | stats min(FreeSpace) as hostavg by host,instance | table host,instance,hostavg | chart min(hostavg) by host,instance

It looks like you have created the following base search - however this only returns raw events and not an table

index=perfmon source="Perfmon:LogicalDisk" counter="% Free Space"

I would split this up as follows

Base search

index=perfmon source="Perfmon:LogicalDisk" counter="% Free Space" |  eval FreeSpace =100-( Value ) | stats min(FreeSpace) as hostavg by host,instance 

Post-processing search

|search host = DMOPWMD1PDDB0* |chart min(hostavg) by host,instance

In many cases, you need to create a temporary stats table in the base search, just to get this to work, even if you would not normally do this in an interactive search. If you base search cannot be easily combined into a single stats table, then you can create multiple base searches. I don't see the code you are using for the search ID's so just in case, it needs to be in this general format

  <search id="BaseSearchName1">
    <query>index=........</query>
    <earliest>-24h</earliest>
    <latest>now</latest>
  </search>
0 Karma

jpolvino
Builder

Might not hurt to physically join the base query in front of the panel's query along with the time tokens. Do away with the base search for now.

Also, consider a debug HTML panel to validate your tokens (remove leading spaces):
< row>
< panel>
< html>
< h1>Debug< /h1>
< p>earliest=$TimeRangePkr.earliest$<br/>latest=$TimeRangePkr.latest$< /p>
< /html>
< /panel>
< /row>

0 Karma

anjibabu
New Member

Hi,

actually we are creating multiple panels in the same dashboard, so wanted to use the search base to avoid the performance issues.

search id query "index=perfmon sourcetype="Perfmon:Memory" counter="% Committed Bytes In Use"
search base query " | search host = DMOPWMD1PDDB0*" timechart perc90(Value) by host limit=0 span=1m"

it is not working:(

0 Karma

jpolvino
Builder

When I run into this issue, here's what I do, and it might work for you.
1. Clone the dashboard.
2. Remove all panels except one that is giving you issues.
3. Hard code the full search and the time range into the panel's search. In other words, use the earliest and latest keywords in the search itself before the first pipe. Example: (main search elements) earliest=-1w@w latest=now | search host=.... Test it.
4. Add your time picker input, remove the earliest and latest from your search, and make the search use your time picker tokens. Test it.
5. Remove the front part of the working search to make your base search, use the time tokens in that search, and make your main search use the base. Test it.

0 Karma

anjibabu
New Member

Thank you @jpolvino.

I have tried the above steps, but no luck, still same issue:(

Thanks,

0 Karma

mayurr98
Super Champion

Can you paste your entire dashboard XML i.e source code?

0 Karma

anjibabu
New Member

DB Performance Clone_4/14

index=perfmon source="Perfmon:LogicalDisk" counter="% Free Space"
$TimeRangePkr.earliest$
$TimeRangePkr.latest$
5m
delay

<query>index=perfmon sourcetype="Perfmon:Memory" counter="% Committed Bytes In Use"</query>
<earliest>$TimeRangePkr.earliest$</earliest>
<latest>$TimeRangePkr.latest$</latest>
<refresh>5m</refresh>
<refreshType>delay</refreshType>


<input type="time" token="TimeRangePkr" searchWhenChanged="true">
  <label>Time Range</label>
  <default>
    <earliest>-60m@m</earliest>
    <latest>now</latest>
  </default>
</input>
<input type="checkbox" token="Global">
  <label></label>
  <choice value="Global">Global</choice>
</input>
<input type="checkbox" token="Reporting">
  <label></label>
  <choice value="Reporting">Reporting</choice>
</input>
 </fieldset>

<panel>
  <chart>
    <title>DISK%</title>
    <search base="Disk1">
      <query>search host = DMOPWMD1PDDB0* | eval FreeSpace =100-( Value )  | stats min(FreeSpace) as hostavg by host,instance | table host,instance,hostavg | chart min(hostavg) by host,instance</query>
    </search>
    <option name="charting.axisLabelsX.majorLabelStyle.overflowMode">ellipsisNone</option>
    <option name="charting.axisLabelsX.majorLabelStyle.rotation">0</option>
    <option name="charting.axisTitleX.text">TIME</option>
    <option name="charting.axisTitleX.visibility">visible</option>
    <option name="charting.axisTitleY.text">HOST</option>
    <option name="charting.axisTitleY.visibility">visible</option>
    <option name="charting.axisTitleY2.visibility">visible</option>
    <option name="charting.axisX.scale">linear</option>
    <option name="charting.axisY.scale">linear</option>
    <option name="charting.axisY2.enabled">0</option>
    <option name="charting.axisY2.scale">inherit</option>
    <option name="charting.chart">column</option>
    <option name="charting.chart.bubbleMaximumSize">50</option>
    <option name="charting.chart.bubbleMinimumSize">10</option>
    <option name="charting.chart.bubbleSizeBy">area</option>
    <option name="charting.chart.nullValueMode">gaps</option>
    <option name="charting.chart.sliceCollapsingThreshold">0.01</option>
    <option name="charting.chart.stackMode">default</option>
    <option name="charting.chart.style">shiny</option>
    <option name="charting.drilldown">all</option>
    <option name="charting.layout.splitSeries">0</option>
    <option name="charting.legend.labelStyle.overflowMode">ellipsisMiddle</option>
    <option name="charting.legend.placement">right</option>
    <option name="refresh.display">progressbar</option>
  </chart>
</panel>
<panel>
  <chart>
    <title>MEMORY%</title>
    <search base="Mem">
      <query>search host=DMOPWMD1PDDB0* | timechart perc95(Value) by host limit=0 span=1m</query>
      </search>
    <option name="charting.axisLabelsX.majorLabelStyle.overflowMode">ellipsisNone</option>
    <option name="charting.axisLabelsX.majorLabelStyle.rotation">0</option>
    <option name="charting.axisTitleX.text">TIME</option>
    <option name="charting.axisTitleX.visibility">visible</option>
    <option name="charting.axisTitleY.text">HOST</option>
    <option name="charting.axisTitleY.visibility">visible</option>
    <option name="charting.axisTitleY2.visibility">visible</option>
    <option name="charting.axisX.scale">linear</option>
    <option name="charting.axisY.scale">linear</option>
    <option name="charting.axisY2.enabled">0</option>
    <option name="charting.axisY2.scale">inherit</option>
    <option name="charting.chart">line</option>
    <option name="charting.chart.bubbleMaximumSize">50</option>
    <option name="charting.chart.bubbleMinimumSize">10</option>
    <option name="charting.chart.bubbleSizeBy">area</option>
    <option name="charting.chart.nullValueMode">gaps</option>
    <option name="charting.chart.sliceCollapsingThreshold">0.01</option>
    <option name="charting.chart.stackMode">default</option>
    <option name="charting.chart.style">shiny</option>
    <option name="charting.layout.splitSeries">0</option>
    <option name="charting.legend.labelStyle.overflowMode">ellipsisMiddle</option>
    <option name="charting.legend.placement">right</option>
    <option name="refresh.display">progressbar</option>
  </chart>
</panel>
0 Karma

anjibabu
New Member
<query>index=perfmon source="Perfmon:LogicalDisk" counter="% Free Space"</query>
<earliest>$TimeRangePkr.earliest$</earliest>
<latest>$TimeRangePkr.latest$</latest>
<refresh>5m</refresh>
<refreshType>delay</refreshType>


<query>index=perfmon sourcetype="Perfmon:Memory" counter="% Committed Bytes In Use"</query>
<earliest>$TimeRangePkr.earliest$</earliest>
<latest>$TimeRangePkr.latest$</latest>
<refresh>5m</refresh>
<refreshType>delay</refreshType>
0 Karma

anjibabu
New Member

could you pls advise

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...