Heavy forwarder not sending syslog to indexer

[johnlzy0408]

Previously, my heavy forwarder is working fine. Able to search from latest logs in my searchhead. But upon testing another app for another SIEM in the heavy forwarder, it has been routing to there since. But after the POC ended, we want to switch back to sending it back to our splunk indexer. 

We remove the app for the SIEM and left with our outputs for this forwarder which is towards the splunk indexer IP. I tried restarting the splunk service in this heavy forwarder but still unable to search those host in the searchhead. 

Is there anything to look out for? 

[James_ACN]

Hello @johnlzy0408 

Have you already checked splunkd.log if there is any ERROR log that might clarify the problem?
Have you checked if the communication between HF and Indexer is OK on port 9997.

Or if Iptables could be blocking this communication?

Please provide more details of the issue.

James \°/

[johnlzy0408]

Hi James, 

Checks on the splunkd comes out this few errors

11-09-2021 22:18:40.901 +0800 WARN TailReader - Could not send data to output queue (parsingQueue), retrying...
11-09-2021 22:18:45.796 +0800 WARN TailReader - Could not send data to output queue (parsingQueue), retrying...
11-09-2021 22:18:50.100 +0800 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_192.168.x.xxx_8089_splunk-forwardersg.ifastfinancial.local_splunk-forwarder02_08A348A2-5753-4968-8FC4-F4477414E56B
11-09-2021 22:18:50.797 +0800 INFO TailReader - Continuing...
11-09-2021 22:18:50.901 +0800 INFO TailReader - ...continuing.
11-09-2021 22:18:54.746 +0800 INFO TcpOutputProc - Found currently active indexer. Connected to idx=192.168.x.xxx:9997, reuse=1.
11-09-2021 22:18:55.956 +0800 WARN TailReader - Could not send data to output queue (parsingQueue), retrying...
11-09-2021 22:19:05.957 +0800 INFO TailReader - ...continuing.
11-09-2021 22:19:10.801 +0800 WARN TailReader - Could not send data to output queue (parsingQueue), retrying...
11-09-2021 22:19:20.802 +0800 INFO TailReader - Continuing...
11-09-2021 22:19:24.638 +0800 INFO TcpOutputProc - Found currently active indexer. Connected to idx=192.168.x.xxx:9997, reuse=1.

Connection from Heavy forwarder to indexer is fine, 

Trying 192.168.x.xx...
Connected to 192.168.x.xxx.
Escape character is '^]'.
^]

11-09-2021 23:30:07.713 +0800 INFO TcpOutputProc - Found currently active indexer. Connected to idx=192.168.x.xxx:9997, reuse=1.
11-09-2021 23:30:37.607 +0800 INFO TcpOutputProc - Found currently active indexer. Connected to idx=192.168.x.xxx:9997, reuse=1.