Getting Data In

Why does checking the status of a forwarder make the forwarder stop running?

AKG1_old1
Builder

Hello,

I have multiple Splunk forwarders in my setup. I am writing a script in which I need to check if Splunk forwarders are up and running.

I am executing this script from a remote machine but I am getting unexpected behavior. Like sometimes checking the status stops the Splunk Forwarder.

Example:

bash$ ./splunkforwarder2/bin/splunk status
splunkd is running (PID: 20365).

bash$ ./splunkforwarder2/bin/splunk status
splunkd 11412 was not running.
Stopping splunk helpers...
                                                           [  OK  ]
Done.
Stopped helpers.
Removing stale pid file... done.

Is there any alternate way in which I can find out what specific forwarder is up and running?

Regards,
Ankit

0 Karma

amrit6109
New Member

I would like to see what Linux script you are using which leads to stopping the forwarder and also you can configure the alert via splunk listing all the hosts if you have admin access you can search _internal logs with the hosts and check for the heartbeat delay .

0 Karma

gcusello
SplunkTrust
SplunkTrust

You could insert the list of your forwarder in a lookup and then run a search like this.

| inputlookup perimeter.csv | eval count=0 | eval host=upper(host)| append [ search index=_internal | eval host= upper(host) | stats count by host ] | stats sum(count) AS total by host

In this way hosts with total=0 are missing
I usually have an alert to check this.
Bye.
Giuseppe

0 Karma
Get Updates on the Splunk Community!

New This Month in Splunk Observability Cloud - Synthetic Monitoring updates, UI ...

This month, we’re delivering several platform, infrastructure, application and digital experience monitoring ...

See your relevant APM services, dashboards, and alerts in one place with the updated ...

As a Splunk Observability user, you have a lot of data you have to manage, prioritize, and troubleshoot on a ...

Best Strategies to Optimize Observability Costs

 Join us on Tuesday, May 6, 2025, at 11 AM PDT / 2 PM EDT for an insightful session on optimizing ...