topic Re: Why does checking the status of a forwarder make the forwarder stop running? in Getting Data In

Why does checking the status of a forwarder make the forwarder stop running?

AKG1_old1 — Wed, 21 Sep 2016 15:47:34 GMT

Hello,

I have multiple Splunk forwarders in my setup. I am writing a script in which I need to check if Splunk forwarders are up and running.

I am executing this script from a remote machine but I am getting unexpected behavior. Like sometimes checking the status stops the Splunk Forwarder.

Example:

bash$ ./splunkforwarder2/bin/splunk status
splunkd is running (PID: 20365).

bash$ ./splunkforwarder2/bin/splunk status
splunkd 11412 was not running.
Stopping splunk helpers...
                                                           [  OK  ]
Done.
Stopped helpers.
Removing stale pid file... done.

Is there any alternate way in which I can find out what specific forwarder is up and running?

Regards,
Ankit

Re: Why does checking the status of a forwarder make the forwarder stop running?

gcusello — Wed, 21 Sep 2016 16:24:28 GMT

You could insert the list of your forwarder in a lookup and then run a search like this.

| inputlookup perimeter.csv | eval count=0 | eval host=upper(host)| append [ search index=_internal | eval host= upper(host) | stats count by host ] | stats sum(count) AS total by host

In this way hosts with total=0 are missing
I usually have an alert to check this.
Bye.
Giuseppe

Re: Why does checking the status of a forwarder make the forwarder stop running?

amrit6109 — Tue, 07 Jan 2020 09:13:15 GMT

I would like to see what Linux script you are using which leads to stopping the forwarder and also you can configure the alert via splunk listing all the hosts if you have admin access you can search _internal logs with the hosts and check for the heartbeat delay .