Reporting

Search head doesnt read from cache and always runs scheduled search

kraveruk
Explorer

Hi,

I used to have splunk running on the single node, and I could see that scheduled searches are not running when user visits the dashboard.
Now having one search head and one indexer, I can see that even though search is scheduled, it has been run at least once every dashboard visit is triggering new search job.

I have made sure I am using saved searches and not inline searches.

I do not know if this is relevant to the fact that DMC (also running on search head) is not showing any licensing history, while search like

index="_internal" source="*metrics.log" per_index_thruput | eval GB=kb/(1024*1024) | timechart span=1d sum(GB) | convert ctime(_time) as timestamp

show license usage.

My long shot guess is that search head does not read from all indexes on indexer or do not send logs ?
I am not sure if this explains why saves searches are not being cached at all ?

Please help
Thanks

0 Karma

woodcock
Esteemed Legend

You should not be using your DMC as a search head for exactly these kinds of reasons (the documentation is clear):

http://docs.splunk.com/Documentation/Splunk/6.2.8/Admin/ConfiguretheMonitoringConsole

Look in section "Which instance should host the console?"

0 Karma

kraveruk
Explorer

This setup has been recommended by splunk pre-sales engineer.

0 Karma

esix_splunk
Splunk Employee
Splunk Employee

Do you have distributed search configured and running? If you run the following search, does it return all of your expected indexers?

index=_internal earliest=-5m@m | stats count by splunk_server
0 Karma

kraveruk
Explorer

yes,

I only have 1 indexer, and I can see it in this search result.

0 Karma

somesoni2
Revered Legend

What is the version of Splunk you're using? Can you post your dashboard code portion where you're running the saved search?

0 Karma

kraveruk
Explorer

I am running 6.3.3

Code for the dash:

<panel>
  <title>Connection diagnosis</title>
  <chart>
    <search ref="Connection diagnosis"></search>
    <option name="charting.axisLabelsX.majorLabelStyle.overflowMode">ellipsisNone</option>
    <option name="charting.axisLabelsX.majorLabelStyle.rotation">0</option>
    <option name="charting.axisTitleX.visibility">visible</option>
    <option name="charting.axisTitleY.visibility">visible</option>
    <option name="charting.axisTitleY2.visibility">visible</option>
    <option name="charting.axisX.scale">linear</option>
    <option name="charting.axisY.scale">linear</option>
    <option name="charting.axisY2.enabled">0</option>
    <option name="charting.axisY2.scale">inherit</option>
    <option name="charting.chart">pie</option>
    <option name="charting.chart.bubbleMaximumSize">50</option>
    <option name="charting.chart.bubbleMinimumSize">10</option>
    <option name="charting.chart.bubbleSizeBy">area</option>
    <option name="charting.chart.nullValueMode">gaps</option>
    <option name="charting.chart.showDataLabels">none</option>
    <option name="charting.chart.sliceCollapsingThreshold">0.01</option>
    <option name="charting.chart.stackMode">default</option>
    <option name="charting.chart.style">shiny</option>
    <option name="charting.drilldown">all</option>
    <option name="charting.layout.splitSeries">0</option>
    <option name="charting.layout.splitSeries.allowIndependentYRanges">0</option>
    <option name="charting.legend.labelStyle.overflowMode">ellipsisMiddle</option>
    <option name="charting.legend.placement">right</option>
    </drilldown>
  </chart>
</panel>
<panel>
  <title>Connection distribution by platform</title>
  <chart>
    <search ref="_CS: Platform breakdown"></search>
    <option name="charting.axisLabelsX.majorLabelStyle.overflowMode">ellipsisNone</option>
    <option name="charting.axisLabelsX.majorLabelStyle.rotation">0</option>
    <option name="charting.axisTitleX.visibility">visible</option>
    <option name="charting.axisTitleY.visibility">visible</option>
    <option name="charting.axisTitleY2.visibility">visible</option>
    <option name="charting.axisX.scale">linear</option>
    <option name="charting.axisY.scale">linear</option>
    <option name="charting.axisY2.enabled">0</option>
    <option name="charting.axisY2.scale">inherit</option>
    <option name="charting.chart">pie</option>
    <option name="charting.chart.bubbleMaximumSize">50</option>
    <option name="charting.chart.bubbleMinimumSize">10</option>
    <option name="charting.chart.bubbleSizeBy">area</option>
    <option name="charting.chart.nullValueMode">gaps</option>
    <option name="charting.chart.rangeValues">[0,30,70,100]</option>
    <option name="charting.chart.showDataLabels">all</option>
    <option name="charting.chart.sliceCollapsingThreshold">0.01</option>
    <option name="charting.chart.stackMode">default</option>
    <option name="charting.chart.style">shiny</option>
    <option name="charting.drilldown">all</option>
    <option name="charting.gaugeColors">["0xd13b3b","0xFFE800","0x7e9f44"]</option>
    <option name="charting.layout.splitSeries">0</option>
    <option name="charting.layout.splitSeries.allowIndependentYRanges">0</option>
    <option name="charting.legend.labelStyle.overflowMode">ellipsisMiddle</option>
    <option name="charting.legend.placement">right</option>
  </chart>
</panel>
0 Karma
Get Updates on the Splunk Community!

Routing Data to Different Splunk Indexes in the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...

Getting Started with AIOps: Event Correlation Basics and Alert Storm Detection in ...

Getting Started with AIOps:Event Correlation Basics and Alert Storm Detection in Splunk IT Service ...

Register to Attend BSides SPL 2022 - It's all Happening October 18!

Join like-minded individuals for technical sessions on everything Splunk!  This is a community-led and run ...