Getting Data In

Forwarder refusing to start

mawomommoh
Path Finder

My forwarder was working fine but stopped and I can't get it running again. Running the splunk start command appears to be working fine but then it fails at the last step.

alt text

No logs are being generated in splunkd.log but some logs are generated in splunkd-utility.log:

alt text

I am not certain what is causing it from starting. Any help would be appreciated. Thanks

0 Karma

renjith_nair
SplunkTrust
SplunkTrust

Hi @mawomommoh,

Check the following,

  • Do you have any crash log in splunk log directory?
  • Directory permissions are intact.
  • System has enough memory.
  • Check in windows events to see if there any issues related to splunk crash
Happy Splunking!
0 Karma

mawomommoh
Path Finder

Okay. I checked my log files again and I can see a bunch of errors and warnings from before I started encountering the issue:

  • Processing server from outpus.conf: can't resolve a valid IP address for host=XXXX
  • Cooked connection to ip=XXXX timed out
  • Connection to XXXX closed. Read error. An existing connection was forcibly closed by remote host.
  • The TCP processor has paused the data flow. Forwarding to output group default-autolb-group has been blocked for 2300 seconds. This will probably stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.

I have checked my splunk server and deleted/created port 9997 which I am using for the forwarding but still no difference.

0 Karma

renjith_nair
SplunkTrust
SplunkTrust

Hi @mawomommoh,

Check your index configuration (host and ip) in output conf of forwarder and make sure that they are reachable and not blocked by firewall or acl

Happy Splunking!
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...