Search head doesnt read from cache and always runs scheduled search

[kraveruk]

Hi,

I used to have splunk running on the single node, and I could see that scheduled searches are not running when user visits the dashboard.
Now having one search head and one indexer, I can see that even though search is scheduled, it has been run at least once every dashboard visit is triggering new search job.

I have made sure I am using saved searches and not inline searches.

I do not know if this is relevant to the fact that DMC (also running on search head) is not showing any licensing history, while search like

index="_internal" source="*metrics.log" per_index_thruput | eval GB=kb/(1024*1024) | timechart span=1d sum(GB) | convert ctime(_time) as timestamp

show license usage.

My long shot guess is that search head does not read from all indexes on indexer or do not send logs ?
I am not sure if this explains why saves searches are not being cached at all ?

Please help
Thanks

[woodcock]

You should not be using your DMC as a search head for exactly these kinds of reasons (the documentation is clear):

http://docs.splunk.com/Documentation/Splunk/6.2.8/Admin/ConfiguretheMonitoringConsole

Look in section "Which instance should host the console?"

[kraveruk]

This setup has been recommended by splunk pre-sales engineer.

[esix_splunk]

Do you have distributed search configured and running? If you run the following search, does it return all of your expected indexers?

index=_internal earliest=-5m@m | stats count by splunk_server

[kraveruk]

yes,

I only have 1 indexer, and I can see it in this search result.

[somesoni2]

What is the version of Splunk you're using? Can you post your dashboard code portion where you're running the saved search?

[kraveruk]

I am running 6.3.3

Code for the dash:

<panel>
  <title>Connection diagnosis</title>
  <chart>
    <search ref="Connection diagnosis"></search>
    <option name="charting.axisLabelsX.majorLabelStyle.overflowMode">ellipsisNone</option>
    <option name="charting.axisLabelsX.majorLabelStyle.rotation">0</option>
    <option name="charting.axisTitleX.visibility">visible</option>
    <option name="charting.axisTitleY.visibility">visible</option>
    <option name="charting.axisTitleY2.visibility">visible</option>
    <option name="charting.axisX.scale">linear</option>
    <option name="charting.axisY.scale">linear</option>
    <option name="charting.axisY2.enabled">0</option>
    <option name="charting.axisY2.scale">inherit</option>
    <option name="charting.chart">pie</option>
    <option name="charting.chart.bubbleMaximumSize">50</option>
    <option name="charting.chart.bubbleMinimumSize">10</option>
    <option name="charting.chart.bubbleSizeBy">area</option>
    <option name="charting.chart.nullValueMode">gaps</option>
    <option name="charting.chart.showDataLabels">none</option>
    <option name="charting.chart.sliceCollapsingThreshold">0.01</option>
    <option name="charting.chart.stackMode">default</option>
    <option name="charting.chart.style">shiny</option>
    <option name="charting.drilldown">all</option>
    <option name="charting.layout.splitSeries">0</option>
    <option name="charting.layout.splitSeries.allowIndependentYRanges">0</option>
    <option name="charting.legend.labelStyle.overflowMode">ellipsisMiddle</option>
    <option name="charting.legend.placement">right</option>
    </drilldown>
  </chart>
</panel>
<panel>
  <title>Connection distribution by platform</title>
  <chart>
    <search ref="_CS: Platform breakdown"></search>
    <option name="charting.axisLabelsX.majorLabelStyle.overflowMode">ellipsisNone</option>
    <option name="charting.axisLabelsX.majorLabelStyle.rotation">0</option>
    <option name="charting.axisTitleX.visibility">visible</option>
    <option name="charting.axisTitleY.visibility">visible</option>
    <option name="charting.axisTitleY2.visibility">visible</option>
    <option name="charting.axisX.scale">linear</option>
    <option name="charting.axisY.scale">linear</option>
    <option name="charting.axisY2.enabled">0</option>
    <option name="charting.axisY2.scale">inherit</option>
    <option name="charting.chart">pie</option>
    <option name="charting.chart.bubbleMaximumSize">50</option>
    <option name="charting.chart.bubbleMinimumSize">10</option>
    <option name="charting.chart.bubbleSizeBy">area</option>
    <option name="charting.chart.nullValueMode">gaps</option>
    <option name="charting.chart.rangeValues">[0,30,70,100]</option>
    <option name="charting.chart.showDataLabels">all</option>
    <option name="charting.chart.sliceCollapsingThreshold">0.01</option>
    <option name="charting.chart.stackMode">default</option>
    <option name="charting.chart.style">shiny</option>
    <option name="charting.drilldown">all</option>
    <option name="charting.gaugeColors">["0xd13b3b","0xFFE800","0x7e9f44"]</option>
    <option name="charting.layout.splitSeries">0</option>
    <option name="charting.layout.splitSeries.allowIndependentYRanges">0</option>
    <option name="charting.legend.labelStyle.overflowMode">ellipsisMiddle</option>
    <option name="charting.legend.placement">right</option>
  </chart>
</panel>