Forwarder refusing to start

[mawomommoh]

My forwarder was working fine but stopped and I can't get it running again. Running the splunk start command appears to be working fine but then it fails at the last step.

No logs are being generated in splunkd.log but some logs are generated in splunkd-utility.log:

I am not certain what is causing it from starting. Any help would be appreciated. Thanks

[renjith_nair]

Hi @mawomommoh,

Check the following,

• Do you have any crash log in splunk log directory?
• Directory permissions are intact.
• System has enough memory.
• Check in windows events to see if there any issues related to splunk crash

[mawomommoh]

Okay. I checked my log files again and I can see a bunch of errors and warnings from before I started encountering the issue:

• Processing server from outpus.conf: can't resolve a valid IP address for host=XXXX
• Cooked connection to ip=XXXX timed out
• Connection to XXXX closed. Read error. An existing connection was forcibly closed by remote host.
• The TCP processor has paused the data flow. Forwarding to output group default-autolb-group has been blocked for 2300 seconds. This will probably stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.

I have checked my splunk server and deleted/created port 9997 which I am using for the forwarding but still no difference.

[renjith_nair]

Hi @mawomommoh,

Check your index configuration (host and ip) in output conf of forwarder and make sure that they are reachable and not blocked by firewall or acl