Why does checking the status of a forwarder make the forwarder stop running?

[AKG1_old1]

Hello,

I have multiple Splunk forwarders in my setup. I am writing a script in which I need to check if Splunk forwarders are up and running.

I am executing this script from a remote machine but I am getting unexpected behavior. Like sometimes checking the status stops the Splunk Forwarder.

Example:

bash$ ./splunkforwarder2/bin/splunk status
splunkd is running (PID: 20365).

bash$ ./splunkforwarder2/bin/splunk status
splunkd 11412 was not running.
Stopping splunk helpers...
                                                           [  OK  ]
Done.
Stopped helpers.
Removing stale pid file... done.

Is there any alternate way in which I can find out what specific forwarder is up and running?

Regards,
Ankit

[amrit6109]

I would like to see what Linux script you are using which leads to stopping the forwarder and also you can configure the alert via splunk listing all the hosts if you have admin access you can search _internal logs with the hosts and check for the heartbeat delay .

[gcusello]

You could insert the list of your forwarder in a lookup and then run a search like this.

| inputlookup perimeter.csv | eval count=0 | eval host=upper(host)| append [ search index=_internal | eval host= upper(host) | stats count by host ] | stats sum(count) AS total by host

In this way hosts with total=0 are missing
I usually have an alert to check this.
Bye.
Giuseppe