Reporting

Search head doesnt read from cache and always runs scheduled search

kraveruk
Explorer

Hi,

I used to have splunk running on the single node, and I could see that scheduled searches are not running when user visits the dashboard.
Now having one search head and one indexer, I can see that even though search is scheduled, it has been run at least once every dashboard visit is triggering new search job.

I have made sure I am using saved searches and not inline searches.

I do not know if this is relevant to the fact that DMC (also running on search head) is not showing any licensing history, while search like

index="_internal" source="*metrics.log" per_index_thruput | eval GB=kb/(1024*1024) | timechart span=1d sum(GB) | convert ctime(_time) as timestamp

show license usage.

My long shot guess is that search head does not read from all indexes on indexer or do not send logs ?
I am not sure if this explains why saves searches are not being cached at all ?

Please help
Thanks

0 Karma

woodcock
Esteemed Legend

You should not be using your DMC as a search head for exactly these kinds of reasons (the documentation is clear):

http://docs.splunk.com/Documentation/Splunk/6.2.8/Admin/ConfiguretheMonitoringConsole

Look in section "Which instance should host the console?"

0 Karma

kraveruk
Explorer

This setup has been recommended by splunk pre-sales engineer.

0 Karma

esix_splunk
Splunk Employee
Splunk Employee

Do you have distributed search configured and running? If you run the following search, does it return all of your expected indexers?

index=_internal earliest=-5m@m | stats count by splunk_server
0 Karma

kraveruk
Explorer

yes,

I only have 1 indexer, and I can see it in this search result.

0 Karma

somesoni2
SplunkTrust
SplunkTrust

What is the version of Splunk you're using? Can you post your dashboard code portion where you're running the saved search?

0 Karma

kraveruk
Explorer

I am running 6.3.3

Code for the dash:

<panel>
  <title>Connection diagnosis</title>
  <chart>
    <search ref="Connection diagnosis"></search>
    <option name="charting.axisLabelsX.majorLabelStyle.overflowMode">ellipsisNone</option>
    <option name="charting.axisLabelsX.majorLabelStyle.rotation">0</option>
    <option name="charting.axisTitleX.visibility">visible</option>
    <option name="charting.axisTitleY.visibility">visible</option>
    <option name="charting.axisTitleY2.visibility">visible</option>
    <option name="charting.axisX.scale">linear</option>
    <option name="charting.axisY.scale">linear</option>
    <option name="charting.axisY2.enabled">0</option>
    <option name="charting.axisY2.scale">inherit</option>
    <option name="charting.chart">pie</option>
    <option name="charting.chart.bubbleMaximumSize">50</option>
    <option name="charting.chart.bubbleMinimumSize">10</option>
    <option name="charting.chart.bubbleSizeBy">area</option>
    <option name="charting.chart.nullValueMode">gaps</option>
    <option name="charting.chart.showDataLabels">none</option>
    <option name="charting.chart.sliceCollapsingThreshold">0.01</option>
    <option name="charting.chart.stackMode">default</option>
    <option name="charting.chart.style">shiny</option>
    <option name="charting.drilldown">all</option>
    <option name="charting.layout.splitSeries">0</option>
    <option name="charting.layout.splitSeries.allowIndependentYRanges">0</option>
    <option name="charting.legend.labelStyle.overflowMode">ellipsisMiddle</option>
    <option name="charting.legend.placement">right</option>
    </drilldown>
  </chart>
</panel>
<panel>
  <title>Connection distribution by platform</title>
  <chart>
    <search ref="_CS: Platform breakdown"></search>
    <option name="charting.axisLabelsX.majorLabelStyle.overflowMode">ellipsisNone</option>
    <option name="charting.axisLabelsX.majorLabelStyle.rotation">0</option>
    <option name="charting.axisTitleX.visibility">visible</option>
    <option name="charting.axisTitleY.visibility">visible</option>
    <option name="charting.axisTitleY2.visibility">visible</option>
    <option name="charting.axisX.scale">linear</option>
    <option name="charting.axisY.scale">linear</option>
    <option name="charting.axisY2.enabled">0</option>
    <option name="charting.axisY2.scale">inherit</option>
    <option name="charting.chart">pie</option>
    <option name="charting.chart.bubbleMaximumSize">50</option>
    <option name="charting.chart.bubbleMinimumSize">10</option>
    <option name="charting.chart.bubbleSizeBy">area</option>
    <option name="charting.chart.nullValueMode">gaps</option>
    <option name="charting.chart.rangeValues">[0,30,70,100]</option>
    <option name="charting.chart.showDataLabels">all</option>
    <option name="charting.chart.sliceCollapsingThreshold">0.01</option>
    <option name="charting.chart.stackMode">default</option>
    <option name="charting.chart.style">shiny</option>
    <option name="charting.drilldown">all</option>
    <option name="charting.gaugeColors">["0xd13b3b","0xFFE800","0x7e9f44"]</option>
    <option name="charting.layout.splitSeries">0</option>
    <option name="charting.layout.splitSeries.allowIndependentYRanges">0</option>
    <option name="charting.legend.labelStyle.overflowMode">ellipsisMiddle</option>
    <option name="charting.legend.placement">right</option>
  </chart>
</panel>
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...