Splunk not receiving logs on heavy forwarder

[splunkkk]

Hi. Recently I notice that the splunk heavy forwarder has stop receiving logs from network devices.  We are using TLS over syslog, but the cert is not expired yet. The rsyslog.conf file should be nothing wrong since previously it can receive logs. Can I know why is it happening?

[splunkkk]

Hi all,

I tried restarting the Splunk service on heavy forwarder and logs are coming in again.

Can I know why does Splunk stop receiving logs suddenly and we need to restart the service for it to work again?

Thanks

[isoutamo]

If i recall correctly there are some versions which could have some issues with ingesting data (at least in UF side, but you have HF). 

The best option to get more information is look you _internal logs and try to get information what has happened when (and just before) this issue has arise. As @livehybrid said try 1st figure out is the issue has been on receive or send side or even an indexers?

https://community.splunk.com/t5/Getting-Data-In/Splunk-Indexer-Parsing-Queue-Blocking/td-p/583312 one old post which could related to this issue or at least it contains some useful links.

[livehybrid]

Hi @splunkkk 

Are you still getting other logs / _internal logs from the HF? This will help determine if the error is with sending or receiving data.

Check the $SPLUNK_HOME/var/log/splunk/splunkd.log for any errors relating to SSL/TLS/input/output/queues

Use netcat to check the expected port is open (nc -vz -w1 localhost <port>) - This assume netcat is installed and as "nc" binary.

Please let me know how you get on and consider adding karma to this or any other answer if it has helped.
Regards

Will

[kiran_panchavat]

@splunkkk 

• Ensure no firewall rules or network policies have changed recently that might block traffic (e.g., port 514 or your custom syslog port).
• Ensure rsyslog is running on the HF (systemctl status rsyslog or service rsyslog status).
• Check the disk space on the Syslog forwarder. command:- df -h

• Verify whether any queues are blocked on the heavy forwarder by running:  tail -n 100 /opt/splunk/var/log/splunk/metrics.log | grep -i "blocked=true"

[splunkkk]

Hi @kiran_panchavat 

Firewall rules are in place, nobody has make changes to it.

Rsyslog is running on HF and disk space should be enough as it can still receive some network devices log on the same HF

Any idea what else I can check? Thanks

[kiran_panchavat]

@splunkkk 

Firstly, could you kindly confirm whether your Syslog forwarder is receiving the network logs?
You can verify this by running a tcpdump capture.

To check for devices from which logs are not being received, please use the following command:

sudo tcpdump -i <interface> host <device_IP> and port <port_number>

Replace <interface>, <device_IP>, and <port_number> with the appropriate values for your environment.

Find Interface Names

tcpdump -D

To capture traffic for a specific host (e.g., 192.168.1.50):

sudo tcpdump -i ens160 host 192.168.1.50 ( change your interface here )

To capture traffic on a specific port (e.g., 514):

sudo tcpdump -i ens160 port 515