Getting Data In

Why does checking the status of a forwarder make the forwarder stop running?



I have multiple Splunk forwarders in my setup. I am writing a script in which I need to check if Splunk forwarders are up and running.

I am executing this script from a remote machine but I am getting unexpected behavior. Like sometimes checking the status stops the Splunk Forwarder.


bash$ ./splunkforwarder2/bin/splunk status
splunkd is running (PID: 20365).

bash$ ./splunkforwarder2/bin/splunk status
splunkd 11412 was not running.
Stopping splunk helpers...
                                                           [  OK  ]
Stopped helpers.
Removing stale pid file... done.

Is there any alternate way in which I can find out what specific forwarder is up and running?


0 Karma

New Member

I would like to see what Linux script you are using which leads to stopping the forwarder and also you can configure the alert via splunk listing all the hosts if you have admin access you can search _internal logs with the hosts and check for the heartbeat delay .

0 Karma


You could insert the list of your forwarder in a lookup and then run a search like this.

| inputlookup perimeter.csv | eval count=0 | eval host=upper(host)| append [ search index=_internal | eval host= upper(host) | stats count by host ] | stats sum(count) AS total by host

In this way hosts with total=0 are missing
I usually have an alert to check this.

0 Karma
Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) | New Releases

In the last month, the Splunk Threat Research Team (STRT) has had 2 releases of new security content via the ...

Announcing the 1st Round Champion’s Tribute Winners of the Great Resilience Quest

We are happy to announce the 20 lucky questers who are selected to be the first round of Champion's Tribute ...

We’ve Got Education Validation!

Are you feeling it? All the career-boosting benefits of up-skilling with Splunk? It’s not just a feeling, it's ...