Reporting
Highlighted

Search head doesnt read from cache and always runs scheduled search

Explorer

Hi,

I used to have splunk running on the single node, and I could see that scheduled searches are not running when user visits the dashboard.
Now having one search head and one indexer, I can see that even though search is scheduled, it has been run at least once every dashboard visit is triggering new search job.

I have made sure I am using saved searches and not inline searches.

I do not know if this is relevant to the fact that DMC (also running on search head) is not showing any licensing history, while search like

index="internal" source="*metrics.log" perindexthruput | eval GB=kb/(1024*1024) | timechart span=1d sum(GB) | convert ctime(time) as timestamp

show license usage.

My long shot guess is that search head does not read from all indexes on indexer or do not send logs ?
I am not sure if this explains why saves searches are not being cached at all ?

Please help
Thanks

0 Karma
Highlighted

Re: Search head doesnt read from cache and always runs scheduled search

SplunkTrust
SplunkTrust

What is the version of Splunk you're using? Can you post your dashboard code portion where you're running the saved search?

0 Karma
Highlighted

Re: Search head doesnt read from cache and always runs scheduled search

Explorer

I am running 6.3.3

Code for the dash:

<panel>
  <title>Connection diagnosis</title>
  <chart>
    <search ref="Connection diagnosis"></search>
    <option name="charting.axisLabelsX.majorLabelStyle.overflowMode">ellipsisNone</option>
    <option name="charting.axisLabelsX.majorLabelStyle.rotation">0</option>
    <option name="charting.axisTitleX.visibility">visible</option>
    <option name="charting.axisTitleY.visibility">visible</option>
    <option name="charting.axisTitleY2.visibility">visible</option>
    <option name="charting.axisX.scale">linear</option>
    <option name="charting.axisY.scale">linear</option>
    <option name="charting.axisY2.enabled">0</option>
    <option name="charting.axisY2.scale">inherit</option>
    <option name="charting.chart">pie</option>
    <option name="charting.chart.bubbleMaximumSize">50</option>
    <option name="charting.chart.bubbleMinimumSize">10</option>
    <option name="charting.chart.bubbleSizeBy">area</option>
    <option name="charting.chart.nullValueMode">gaps</option>
    <option name="charting.chart.showDataLabels">none</option>
    <option name="charting.chart.sliceCollapsingThreshold">0.01</option>
    <option name="charting.chart.stackMode">default</option>
    <option name="charting.chart.style">shiny</option>
    <option name="charting.drilldown">all</option>
    <option name="charting.layout.splitSeries">0</option>
    <option name="charting.layout.splitSeries.allowIndependentYRanges">0</option>
    <option name="charting.legend.labelStyle.overflowMode">ellipsisMiddle</option>
    <option name="charting.legend.placement">right</option>
    </drilldown>
  </chart>
</panel>
<panel>
  <title>Connection distribution by platform</title>
  <chart>
    <search ref="_CS: Platform breakdown"></search>
    <option name="charting.axisLabelsX.majorLabelStyle.overflowMode">ellipsisNone</option>
    <option name="charting.axisLabelsX.majorLabelStyle.rotation">0</option>
    <option name="charting.axisTitleX.visibility">visible</option>
    <option name="charting.axisTitleY.visibility">visible</option>
    <option name="charting.axisTitleY2.visibility">visible</option>
    <option name="charting.axisX.scale">linear</option>
    <option name="charting.axisY.scale">linear</option>
    <option name="charting.axisY2.enabled">0</option>
    <option name="charting.axisY2.scale">inherit</option>
    <option name="charting.chart">pie</option>
    <option name="charting.chart.bubbleMaximumSize">50</option>
    <option name="charting.chart.bubbleMinimumSize">10</option>
    <option name="charting.chart.bubbleSizeBy">area</option>
    <option name="charting.chart.nullValueMode">gaps</option>
    <option name="charting.chart.rangeValues">[0,30,70,100]</option>
    <option name="charting.chart.showDataLabels">all</option>
    <option name="charting.chart.sliceCollapsingThreshold">0.01</option>
    <option name="charting.chart.stackMode">default</option>
    <option name="charting.chart.style">shiny</option>
    <option name="charting.drilldown">all</option>
    <option name="charting.gaugeColors">["0xd13b3b","0xFFE800","0x7e9f44"]</option>
    <option name="charting.layout.splitSeries">0</option>
    <option name="charting.layout.splitSeries.allowIndependentYRanges">0</option>
    <option name="charting.legend.labelStyle.overflowMode">ellipsisMiddle</option>
    <option name="charting.legend.placement">right</option>
  </chart>
</panel>
0 Karma
Highlighted

Re: Search head doesnt read from cache and always runs scheduled search

Splunk Employee
Splunk Employee

Do you have distributed search configured and running? If you run the following search, does it return all of your expected indexers?

index=_internal earliest=-5m@m | stats count by splunk_server
0 Karma
Highlighted

Re: Search head doesnt read from cache and always runs scheduled search

Explorer

yes,

I only have 1 indexer, and I can see it in this search result.

0 Karma
Highlighted

Re: Search head doesnt read from cache and always runs scheduled search

Esteemed Legend

You should not be using your DMC as a search head for exactly these kinds of reasons (the documentation is clear):

http://docs.splunk.com/Documentation/Splunk/6.2.8/Admin/ConfiguretheMonitoringConsole

Look in section "Which instance should host the console?"

0 Karma
Highlighted

Re: Search head doesnt read from cache and always runs scheduled search

Explorer

This setup has been recommended by splunk pre-sales engineer.

0 Karma