topic Re: How to determine how long splunk has been up? in Monitoring Splunk

How to determine how long splunk has been up?

wrangler2x — Tue, 08 Oct 2013 15:52:44 GMT

Is there a command in splunk or some way to find out how long it has been up since the last restart?

Re: How to determine how long splunk has been up?

lguinn2 — Tue, 08 Oct 2013 16:00:50 GMT

Try this search:

index=_internal "splunkd started"

to find out when was the last time that splunkd was started. Note that you may have to also add host=zzzz if you want to restrict to a particular host.

If you really want only the uptime, try this:

index=_internal "splunkd started"
| head 1
| eval uptime = tostring(now() - _time,"duration")
| fields uptime

Re: How to determine how long splunk has been up?

kristian_kolb — Tue, 08 Oct 2013 16:08:33 GMT

Searching in sourcetype=splunkd index=_internal you will find a message like this;

10-08-2013 08:55:27.844 +0200 INFO  loader - Splunkd starting (build 143156).

NB, this is for version 5.x, don't know if it differs in 6.x

Re: How to determine how long splunk has been up?

sowings — Tue, 08 Oct 2013 16:16:29 GMT

6.0: 10-07-2013 08:33:05.380 -0700 INFO loader - Splunkd starting (build 182037).

Re: How to determine how long splunk has been up?

tiny3001 — Wed, 26 Oct 2016 11:42:50 GMT

I know I'm ressurecting an old question, but the search is useful.

Except for one thing...

If you don't exclude a specific sourcetype, you get results for your searches looking for "splunkd started". Which might confuse things.
So

index=_internal "splunkd started" NOT sourcetype=splunkd_remote_searches

Hope that helps someone.

Re: How to determine how long splunk has been up?

wrangler2x — Tue, 29 Nov 2016 00:41:03 GMT

You can use the rest API to get this information. Try this:

| rest /services/server/info | eval LastStartupTime=strftime(startup_time, "%Y/%m/%d  %H:%M:%S")
| eval timenow=now()
| eval daysup = round((timenow - startup_time) / 86400,0)
| eval Uptime = tostring(daysup) + " Days"
| table splunk_server LastStartupTime Uptime

Re: How to determine how long splunk has been up?

Sayanta_Basak_I — Wed, 14 Jun 2017 19:41:21 GMT

I downvoted this post because did not work

Re: How to determine how long splunk has been up?

richgalloway — Mon, 27 Nov 2017 17:08:58 GMT

This is the answer that should be accepted, IMO, @wrangler2x. The others, especially the one from @tiny3301, work, but only if Splunk was restarted recently. Once the logs have rolled enough times, the "splunkd started" message won't be found.

Re: How to determine how long splunk has been up?

dijikul — Mon, 10 Sep 2018 18:32:40 GMT

| rest / services/server/info

This only shows indexers. What's the REST endpoint for startup time of all Universal Forwarders?

Re: How to determine how long splunk has been up?

dijikul — Mon, 10 Sep 2018 18:48:16 GMT

This only works when your logs stretch far enough back to catch the startup.

If your forwarders stay online long enough, the logs roll and you lose the data, which is why the REST approach is supposedly better, however I'm having trouble making that work in our Hybrid environment, personally.

Re: How to determine how long splunk has been up?

woodcock — Sun, 02 Jun 2019 23:36:23 GMT

It also did not work because it had | field instead of | fields but I just fixed that.

Re: How to determine how long splunk has been up?

woodcock — Sun, 02 Jun 2019 23:38:37 GMT

This is a refinement of the answers by @lguinn2 and @tiny3001 (NOTE: you will have to edit the host= part):

index=_internal "splunkd started" AND NOT sourcetype=splunkd_remote_searches AND host=*-spl-*
| dedup host
| eval uptime = tostring(now() - _time,"duration")
| table host uptime

Re: How to determine how long splunk has been up?

lakromani — Sun, 25 Oct 2020 18:34:14 GMT

For this to work, you need to set time long enough to catch the restart. With a big solution with many server and lots of logs, this will be slow to find.

Re: How to determine how long splunk has been up?

isoutamo — Sun, 25 Oct 2020 20:30:24 GMT

You probably need to extend the retention time for _internal to get those events stored enough long to find them. In general case the rest is better for full splunk enterprise instances. Of course this requires that you haven’t disable rest on HF layer. For UF the only solution is store those events to internal for enough long time.

Re: How to determine how long splunk has been up?

robfrey — Tue, 12 Apr 2022 01:25:31 GMT

Just happened to be looking for this very thing today and stumbled across submission. I wouldn't have thought to query the REST API for this without checking here first, but it seemed a little obvious after reading the accepted solution -- that's what I love about strong user communities.

For what it's worth, here's my own slightly more direct SPL that produces roughly the same results as the accepted answer in case it helps anyone else.

| rest splunk_server=local /services/server/info 
| eval uptime=tostring(now() - startup_time, "duration") 
| convert ctime(startup_time) 
| table splunk_server, startup_time, uptime

Re: How to determine how long splunk has been up?

robfrey — Tue, 12 Apr 2022 01:30:43 GMT

I should have included that the SPL I provided is only searching the REST API of the search head executing the search (splunk_server=localhost) which can easily be removed if desired

Re: How to determine how long splunk has been up?

BDein — Wed, 11 Oct 2023 10:54:35 GMT

Try this one:

index=_internal "Splunkd starting" sourcetype=splunkd component=loader AND host=* | append [| search index=_internal "splunkd started" sourcetype=splunkd_stderr AND host=* ] | eval st_{sourcetype}=1 | stats count sum(st_*) AS * earliest(_time) AS firstTime latest(_time) AS lastTime BY host | eval uptime = tostring(now() - lastTime,"duration") | foreach *Time [| eval <<FIELD>>=strftime(<<FIELD>>,"%Y-%m-%d %H:%M:%S") ] | table host count firstTime lastTime uptime *

Re: How to determine how long splunk has been up?

isoutamo — Fri, 13 Oct 2023 10:28:34 GMT

As it has said earlier all queries from _internal logs works only if you have those still on indexers. Quite often retention time for those is so short that you haven't those on any larger environment!