topic Re: Forwarder refusing to start in Getting Data In

Forwarder refusing to start

mawomommoh — Sat, 16 Jun 2018 03:47:19 GMT

My forwarder was working fine but stopped and I can't get it running again. Running the splunk start command appears to be working fine but then it fails at the last step.

No logs are being generated in splunkd.log but some logs are generated in splunkd-utility.log:

I am not certain what is causing it from starting. Any help would be appreciated. Thanks

Re: Forwarder refusing to start

renjith_nair — Sat, 16 Jun 2018 04:22:03 GMT

Hi @mawomommoh,

Check the following,

Do you have any crash log in splunk log directory?
Directory permissions are intact.
System has enough memory.
Check in windows events to see if there any issues related to splunk crash

Re: Forwarder refusing to start

mawomommoh — Tue, 19 Jun 2018 09:00:14 GMT

Okay. I checked my log files again and I can see a bunch of errors and warnings from before I started encountering the issue:

Processing server from outpus.conf: can't resolve a valid IP address for host=XXXX
Cooked connection to ip=XXXX timed out
Connection to XXXX closed. Read error. An existing connection was forcibly closed by remote host.
The TCP processor has paused the data flow. Forwarding to output group default-autolb-group has been blocked for 2300 seconds. This will probably stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.

I have checked my splunk server and deleted/created port 9997 which I am using for the forwarding but still no difference.

Re: Forwarder refusing to start

renjith_nair — Wed, 20 Jun 2018 08:43:42 GMT

Hi @mawomommoh,

Check your index configuration (host and ip) in output conf of forwarder and make sure that they are reachable and not blocked by firewall or acl