Reporting

Search head doesnt read from cache and always runs scheduled search

kraveruk
Explorer

Hi,

I used to have splunk running on the single node, and I could see that scheduled searches are not running when user visits the dashboard.
Now having one search head and one indexer, I can see that even though search is scheduled, it has been run at least once every dashboard visit is triggering new search job.

I have made sure I am using saved searches and not inline searches.

I do not know if this is relevant to the fact that DMC (also running on search head) is not showing any licensing history, while search like

index="_internal" source="*metrics.log" per_index_thruput | eval GB=kb/(1024*1024) | timechart span=1d sum(GB) | convert ctime(_time) as timestamp

show license usage.

My long shot guess is that search head does not read from all indexes on indexer or do not send logs ?
I am not sure if this explains why saves searches are not being cached at all ?

Please help
Thanks

0 Karma

woodcock
Esteemed Legend

You should not be using your DMC as a search head for exactly these kinds of reasons (the documentation is clear):

http://docs.splunk.com/Documentation/Splunk/6.2.8/Admin/ConfiguretheMonitoringConsole

Look in section "Which instance should host the console?"

0 Karma

kraveruk
Explorer

This setup has been recommended by splunk pre-sales engineer.

0 Karma

esix_splunk
Splunk Employee
Splunk Employee

Do you have distributed search configured and running? If you run the following search, does it return all of your expected indexers?

index=_internal earliest=-5m@m | stats count by splunk_server
0 Karma

kraveruk
Explorer

yes,

I only have 1 indexer, and I can see it in this search result.

0 Karma

somesoni2
SplunkTrust
SplunkTrust

What is the version of Splunk you're using? Can you post your dashboard code portion where you're running the saved search?

0 Karma

kraveruk
Explorer

I am running 6.3.3

Code for the dash:

<panel>
  <title>Connection diagnosis</title>
  <chart>
    <search ref="Connection diagnosis"></search>
    <option name="charting.axisLabelsX.majorLabelStyle.overflowMode">ellipsisNone</option>
    <option name="charting.axisLabelsX.majorLabelStyle.rotation">0</option>
    <option name="charting.axisTitleX.visibility">visible</option>
    <option name="charting.axisTitleY.visibility">visible</option>
    <option name="charting.axisTitleY2.visibility">visible</option>
    <option name="charting.axisX.scale">linear</option>
    <option name="charting.axisY.scale">linear</option>
    <option name="charting.axisY2.enabled">0</option>
    <option name="charting.axisY2.scale">inherit</option>
    <option name="charting.chart">pie</option>
    <option name="charting.chart.bubbleMaximumSize">50</option>
    <option name="charting.chart.bubbleMinimumSize">10</option>
    <option name="charting.chart.bubbleSizeBy">area</option>
    <option name="charting.chart.nullValueMode">gaps</option>
    <option name="charting.chart.showDataLabels">none</option>
    <option name="charting.chart.sliceCollapsingThreshold">0.01</option>
    <option name="charting.chart.stackMode">default</option>
    <option name="charting.chart.style">shiny</option>
    <option name="charting.drilldown">all</option>
    <option name="charting.layout.splitSeries">0</option>
    <option name="charting.layout.splitSeries.allowIndependentYRanges">0</option>
    <option name="charting.legend.labelStyle.overflowMode">ellipsisMiddle</option>
    <option name="charting.legend.placement">right</option>
    </drilldown>
  </chart>
</panel>
<panel>
  <title>Connection distribution by platform</title>
  <chart>
    <search ref="_CS: Platform breakdown"></search>
    <option name="charting.axisLabelsX.majorLabelStyle.overflowMode">ellipsisNone</option>
    <option name="charting.axisLabelsX.majorLabelStyle.rotation">0</option>
    <option name="charting.axisTitleX.visibility">visible</option>
    <option name="charting.axisTitleY.visibility">visible</option>
    <option name="charting.axisTitleY2.visibility">visible</option>
    <option name="charting.axisX.scale">linear</option>
    <option name="charting.axisY.scale">linear</option>
    <option name="charting.axisY2.enabled">0</option>
    <option name="charting.axisY2.scale">inherit</option>
    <option name="charting.chart">pie</option>
    <option name="charting.chart.bubbleMaximumSize">50</option>
    <option name="charting.chart.bubbleMinimumSize">10</option>
    <option name="charting.chart.bubbleSizeBy">area</option>
    <option name="charting.chart.nullValueMode">gaps</option>
    <option name="charting.chart.rangeValues">[0,30,70,100]</option>
    <option name="charting.chart.showDataLabels">all</option>
    <option name="charting.chart.sliceCollapsingThreshold">0.01</option>
    <option name="charting.chart.stackMode">default</option>
    <option name="charting.chart.style">shiny</option>
    <option name="charting.drilldown">all</option>
    <option name="charting.gaugeColors">["0xd13b3b","0xFFE800","0x7e9f44"]</option>
    <option name="charting.layout.splitSeries">0</option>
    <option name="charting.layout.splitSeries.allowIndependentYRanges">0</option>
    <option name="charting.legend.labelStyle.overflowMode">ellipsisMiddle</option>
    <option name="charting.legend.placement">right</option>
  </chart>
</panel>
0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...