topic Re: Query for Splunkd Status in Splunk Search

Query for Splunkd Status

wvalente — Fri, 24 Nov 2017 13:34:46 GMT

Hi Guys,

There's any query in the splunk web that I'm able to see if the splunkd is not running in a forwarder?

Tks.

Re: Query for Splunkd Status

gcusello — Fri, 24 Nov 2017 14:47:21 GMT

Hi wvalente,
you can check if your forwarders are sending logs to Splunk, reasons for this situations could be many (splunkd not running, network problems, host down, etc...) but anyway I think that you should check if a forwarder is sending logs and not if slunkd is running, anyway if there are other problems you cannot receive logs from the forwarder!

So, to check if a forwarder is sending logs you can search on _internal

index=_internal host=your_host

if you don't receive logs there's a problem.

You can create an alert using a lookup (calling e.g. perimeter.csv) containing all the hosts to monitor and running e.g. every 5 minutes the following search:

index=_internal
| eval host=upper(host)
| stats count by host
| append [ 
     | inputlookup perimeter.csv
     | eval host=upper(host), count=0
     | fields host count
     ]
| stats sum(count) AS Total by host
| where Total=0

In this way, hosts where Total=0 are missing and hosts where Total>0 are sending logs.
You can also show host status in a dashboard (also in graphic mode).

Bye.
Giuseppe

Re: Query for Splunkd Status

niketn — Sat, 25 Nov 2017 16:51:53 GMT

@wvalente, there could be several ways to identify this.

1) By _internal index has already been called out by Giuseppe. For remaining like:
2) metadata command
3) REST api with lastPhoneHomeTime

| rest /services/deployment/server/clients

(https://docs.splunk.com/Documentation/Splunk/latest/RESTREF/RESTdeploy#deployment.2Fserver.2Fclients)
4) Distributed Monitoring Console to monitor deployment status, which uses above REST API.

Refer to answer by @DalJeanis which compiles all these and more 🙂
https://answers.splunk.com/answers/525926/how-do-we-determine-whether-a-forwarder-phoned-hom.html

Re: Query for Splunkd Status

woodcock — Sun, 26 Nov 2017 22:49:06 GMT

I would use tstats like this:

| tstats count max(_indextime) AS _time WHERE index="_*" AND host="YourHostHere"

You can build an alert from there.

Re: Query for Splunkd Status

wvalente — Mon, 27 Nov 2017 13:25:35 GMT

Tks @woodcock, @cusello and @niketnilay

I'll try this solutions.