Splunkforwarder missing

[pacifikn]

Dear All,

Kindly help me am getting error where I find all my splunkforwarder is missing and it shows me it last connected to indexers 02/05/2021.

What causes this? how can I fix this?   

Thank you in advance!!

[gcusello]

Hi @pacifikn,

let me understand:

• you have a forwarder that was running until 02/05/2021,
• then after 02/05/2021 you didn't received any log,

is this correct?

At first, what's the operative system?

Then, did you checked if there's some firewall problem, using telnet?

telnet ip_splunk_server 9997

if telnet is ok, did you checked that the Forwarder is up?

you can do this in this way on Linux:

cd /opt/splunkforwarder/bin
./splunk status

if status is running and telnet is OK, check if the outputs.conf is correctly configured:

• go in /opt/splunkforwarder/system/local,
• edit outputs.conf,
• and check if the configured indexer is the correct one.

At least, are you using Deployment Server to deploy configuratons?

if yes, check that your Forwarder is in the correct ServerClasses.

Ciao.

Giuseppe

[pacifikn]

@gcusello , Thank you so much for your prompt response, but Telnet is disabled. Can you please help me and share me the command(CLI) to check if the firewall is not blocking any service for port:9997

I use Centos 6.4, also kindly help me with the command to use to upgrade my Centos OS version to the latest, Thank you in advance

[gcusello]

Hi @pacifikn,

all Splunk configurations are only in the $SPLUNK_HOME/etc.

you can locate your outputs.conf using a simple Unix command or using btool (https://docs.splunk.com/Documentation/Splunk/8.2.2/Troubleshooting/Usebtooltotroubleshootconfigurati...).

Anyway, outputs.conf is usually in %SPLUNK_HOME/etc/system/local or, if you are using a dedicated app, in %SPLUNK_HOME/etc/apps/<your_app>/local or %SPLUNK_HOME/etc/apps/<your_app>/default.

You can do this check, if you're using a Deployment Server, seeing (or consulting your project documentation) if you deployed an app containing outputs.conf, you can check this in Deployment Server in the folder $SPLUNK_HOME/etc/deployment_apps.

About telnet, it's the easiest way to check if the route to the Indexer is open, if you cannot use telnet, you have to find another way to check this: e.g. verifying routes on intermediate Firewalls.

Ciao.

Giuseppe

[Richfez]

Well, this is really a linux question, not a splunk question.

You should be able to find a forwarder's installed splunk executable by a command similar to

$ sudo find / -name splunk -type f

Look for a line like /opt/splunk/bin/splunk or /opt/splunkforwarder/bin/splunk.

And come to think of it, it's a rarity that it's outside of those two paths in a small environment like what yours sounds like.

As to why it's not running?  That's a question with a zillion different answers.

Possibly it's just not running (e.g. crashed, not set to autostart and the system got rebooted, someone issued a "splunk stop" command, etc...) and you can just start it up.

Possibly it's not able to connect to your indexers  (upgrades to idx and the forward is just too old to talk to it, networking changes, gremlins.

Maybe someone uninstalled the entire forwarder from it.  Disabled it.  Deleted the user it runs as.

As mentioned, the reasons it's not running are nearly infinite, you'll have to just do a little basic troubleshooting.