topic Re: Splunk query to determine how long a Splunk instance was down in past? in Splunk Search

Splunk query to determine how long a Splunk instance was down in past?

saurabh0912 — Thu, 02 Apr 2020 15:22:51 GMT

Hi,
We need to provide report, where we need to capture how long Splunk instance was down in past.
Is it possible to capture using internal logs? What Splunk query can we use to get the duration?

Note: Currently Splunk instances are up and running.

Re: Splunk query to determine how long a Splunk instance was down in past?

richgalloway — Thu, 02 Apr 2020 15:59:28 GMT

You can search the splunkd.log files for "Shutdown complete" and "Splunkd starting" then calculate the difference between those events.

Re: Splunk query to determine how long a Splunk instance was down in past?

saurabh0912 — Thu, 02 Apr 2020 16:13:45 GMT

I just see "splunkd starting" but there are no events for "shutdown complete", what does that mean?

Re: Splunk query to determine how long a Splunk instance was down in past?

richgalloway — Thu, 02 Apr 2020 17:15:15 GMT

I've seen that when Splunk is restarted from the GUI. In that case, there is no indication of the restart. I don't have a solution for that case.

Re: Splunk query to determine how long a Splunk instance was down in past?

PavelP — Fri, 03 Apr 2020 05:44:32 GMT

Splink writes every 10 seconds in the resource_usage.log. With this query you can find gaps in the logging which can indicate when the splunk process was down. This is only an estimation, you have to add/substract the time splunk need to start/shutdown.

index=_introspection sourcetype=splunk_resource_usage

If other splunk instances send internal logs to the indexing layer (always the best practice) then you can find the "downtime" for other splunk instances by specifying the host:

index=_introspection sourcetype=splunk_resource_usage host=XXX

Re: Splunk query to determine how long a Splunk instance was down in past?

woodcock — Fri, 03 Apr 2020 12:12:52 GMT

By default, Splunk only stores the _* logs for 30 days so if you need to go farther back than that, you can infer an outage by looking for a large jump in latency as defined as _time subtracted from_indextime.