Query for Splunkd Status

[wvalente]

Hi Guys,

There's any query in the splunk web that I'm able to see if the splunkd is not running in a forwarder?

Tks.

[wvalente]

Tks @woodcock, @cusello and @niketnilay

I'll try this solutions.

[woodcock]

I would use tstats like this:

| tstats count max(_indextime) AS _time WHERE index="_*" AND host="YourHostHere"

You can build an alert from there.

[niketn]

@wvalente, there could be several ways to identify this.

1) By _internal index has already been called out by Giuseppe. For remaining like:
2) metadata command
3) REST api with lastPhoneHomeTime

| rest /services/deployment/server/clients

(https://docs.splunk.com/Documentation/Splunk/latest/RESTREF/RESTdeploy#deployment.2Fserver.2Fclients)
4) Distributed Monitoring Console to monitor deployment status, which uses above REST API.

Refer to answer by @DalJeanis which compiles all these and more 🙂
https://answers.splunk.com/answers/525926/how-do-we-determine-whether-a-forwarder-phoned-hom.html

[gcusello]

Hi wvalente,
you can check if your forwarders are sending logs to Splunk, reasons for this situations could be many (splunkd not running, network problems, host down, etc...) but anyway I think that you should check if a forwarder is sending logs and not if slunkd is running, anyway if there are other problems you cannot receive logs from the forwarder!

So, to check if a forwarder is sending logs you can search on _internal

index=_internal host=your_host

if you don't receive logs there's a problem.

You can create an alert using a lookup (calling e.g. perimeter.csv) containing all the hosts to monitor and running e.g. every 5 minutes the following search:

index=_internal
| eval host=upper(host)
| stats count by host
| append [ 
     | inputlookup perimeter.csv
     | eval host=upper(host), count=0
     | fields host count
     ]
| stats sum(count) AS Total by host
| where Total=0

In this way, hosts where Total=0 are missing and hosts where Total>0 are sending logs.
You can also show host status in a dashboard (also in graphic mode).

Bye.
Giuseppe