Splunk Restart Tracking

[mbasharat]

Hi folks,

I want to setup a dashboard to track Splunk activities. I need to know how to track who restarted Splunk via both UI and audit logs in Splunk Dashboard?

Thanks in-advance.

[bandit]

Splunk Restart Dashboard

<form theme="dark">
  <label>Splunk Restarts</label>
  <fieldset submitButton="false">
    <input type="time" token="time" searchWhenChanged="true">
      <label>Time Range</label>
      <default>
        <earliest>-24h@h</earliest>
        <latest>now</latest>
      </default>
    </input>
    <input type="text" token="host_include_pattern" searchWhenChanged="true">
      <label>Host Include Pattern (host1,host2)</label>
      <initialValue>*</initialValue>
    </input>
    <input type="text" token="host_exclude_pattern" searchWhenChanged="true">
      <label>Host Exclude Pattern</label>
      <default>null</default>
      <prefix>NOT host="*</prefix>
      <suffix>*"</suffix>
    </input>
  </fieldset>
  <row>
    <panel>
      <title>Unique Instance Restarts</title>
      <single>
        <search>
          <query>index=_internal sourcetype=splunkd source="*splunkd.log" "Splunkd starting" host IN ($host_include_pattern$) $host_exclude_pattern$ 
|  stats dc(host)</query>
          <earliest>$time.earliest$</earliest>
          <latest>$time.latest$</latest>
          <sampleRatio>1</sampleRatio>
          <refresh>30s</refresh>
          <refreshType>delay</refreshType>
        </search>
        <option name="colorMode">block</option>
        <option name="drilldown">none</option>
        <option name="rangeColors">["0x6db7c6","0x6db7c6"]</option>
        <option name="rangeValues">[0]</option>
        <option name="refresh.display">progressbar</option>
        <option name="trellis.enabled">0</option>
        <option name="trellis.scales.shared">1</option>
        <option name="trellis.size">medium</option>
        <option name="useColors">1</option>
      </single>
    </panel>
  </row>
  <row>
    <panel>
      <title>Host Restart Timeline</title>
      <chart>
        <search>
          <query>index=_internal sourcetype=splunkd source="*splunkd.log" "Splunkd starting" host IN ($host_include_pattern$) $host_exclude_pattern$ 
| timechart limit=100 count by host</query>
          <earliest>$time.earliest$</earliest>
          <latest>$time.latest$</latest>
          <sampleRatio>1</sampleRatio>
          <refresh>30s</refresh>
          <refreshType>delay</refreshType>
        </search>
        <option name="charting.axisLabelsX.majorLabelStyle.overflowMode">ellipsisNone</option>
        <option name="charting.axisLabelsX.majorLabelStyle.rotation">0</option>
        <option name="charting.axisTitleX.visibility">visible</option>
        <option name="charting.axisTitleY.visibility">collapsed</option>
        <option name="charting.axisTitleY2.visibility">visible</option>
        <option name="charting.axisX.scale">linear</option>
        <option name="charting.axisY.scale">linear</option>
        <option name="charting.axisY2.enabled">0</option>
        <option name="charting.axisY2.scale">inherit</option>
        <option name="charting.chart">column</option>
        <option name="charting.chart.bubbleMaximumSize">50</option>
        <option name="charting.chart.bubbleMinimumSize">10</option>
        <option name="charting.chart.bubbleSizeBy">area</option>
        <option name="charting.chart.nullValueMode">gaps</option>
        <option name="charting.chart.showDataLabels">none</option>
        <option name="charting.chart.sliceCollapsingThreshold">0.01</option>
        <option name="charting.chart.stackMode">stacked</option>
        <option name="charting.chart.style">shiny</option>
        <option name="charting.drilldown">none</option>
        <option name="charting.layout.splitSeries">0</option>
        <option name="charting.layout.splitSeries.allowIndependentYRanges">0</option>
        <option name="charting.legend.labelStyle.overflowMode">ellipsisMiddle</option>
        <option name="charting.legend.placement">bottom</option>
        <option name="refresh.display">progressbar</option>
        <option name="trellis.enabled">0</option>
        <option name="trellis.scales.shared">1</option>
        <option name="trellis.size">medium</option>
      </chart>
    </panel>
  </row>
  <row>
    <panel>
      <title>Events</title>
      <table>
        <search>
          <query>index=_internal sourcetype=splunkd source="*splunkd.log" "Splunkd starting" host IN ($host_include_pattern$) $host_exclude_pattern$ 
| bucket _time span=5m 
| table _time host _raw 
| sort -_time 
| transaction _time 
| table _time host _raw</query>
          <earliest>$time.earliest$</earliest>
          <latest>$time.latest$</latest>
          <sampleRatio>1</sampleRatio>
          <refresh>30s</refresh>
          <refreshType>delay</refreshType>
        </search>
        <option name="count">20</option>
        <option name="drilldown">none</option>
        <option name="refresh.display">progressbar</option>
      </table>
    </panel>
  </row>
</form>

[_smp_]

Nice. Thanks!

[PowerPacked · Accepted solution]

Hi @mbasharat

Please use these searches

index=_audit action=restart_splunkd | stats c by user
index=_audit action=splunkStarting
index=_audit action=splunkShuttingDown

for UI restart its writes in Splunkd_stdout.log

index=_internal sourcetype=splunkd_stdout

Thanks

[nick405060]

Thanks, these queries all work

except for

splunkShuttingDown

which is not a thing, at least in 7.2.0

[mbasharat]

THANK YOU!!

[adonio]

hello there,

check the _internal index for "splunkd started" or "(build"
many answers here about that, here are couple examples:
https://answers.splunk.com/answers/242618/how-to-count-the-number-of-times-splunk-is-restart.html
https://answers.splunk.com/answers/105128/how-to-determine-how-long-splunk-has-been-up.html

hope it helps

[mbasharat]

Thanks Adonio, is there a way to populate them in a clean list from events. I see my own name in that event and I never started it!