Getting Data In

Why does checking the status of a forwarder make the forwarder stop running?

agoyal
Builder

Hello,

I have multiple Splunk forwarders in my setup. I am writing a script in which I need to check if Splunk forwarders are up and running.

I am executing this script from a remote machine but I am getting unexpected behavior. Like sometimes checking the status stops the Splunk Forwarder.

Example:

bash$ ./splunkforwarder2/bin/splunk status
splunkd is running (PID: 20365).

bash$ ./splunkforwarder2/bin/splunk status
splunkd 11412 was not running.
Stopping splunk helpers...
                                                           [  OK  ]
Done.
Stopped helpers.
Removing stale pid file... done.

Is there any alternate way in which I can find out what specific forwarder is up and running?

Regards,
Ankit

0 Karma

amrit6109
New Member

I would like to see what Linux script you are using which leads to stopping the forwarder and also you can configure the alert via splunk listing all the hosts if you have admin access you can search _internal logs with the hosts and check for the heartbeat delay .

0 Karma

gcusello
Legend

You could insert the list of your forwarder in a lookup and then run a search like this.

| inputlookup perimeter.csv | eval count=0 | eval host=upper(host)| append [ search index=_internal | eval host= upper(host) | stats count by host ] | stats sum(count) AS total by host

In this way hosts with total=0 are missing
I usually have an alert to check this.
Bye.
Giuseppe

0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!