Heavy forwarder not sending logs (Windows)

hkizuka — Fri, 27 Oct 2017 06:18:51 GMT

I've got an issue with HF not sending the logs to indexer.
Does anyone have experience with something like this?

HF was sending the log to indexer as it should until yesterday.
at one moment, indexer OS somehow got shutdown and HF didn't send any logs including internal logs even after the indexer was booted and connection was established.

HF:Windows Server 2012
indexer:Windows Server 2016
Splunk version : 6.6.3

when I checked splunkd.log in HF, I saw logs written as below

10-27-2017 09:07:18.938 +0900 WARN TcpOutputProc - Tcpout Processor: The TCP output processor has paused the data flow. Forwarding to output group splunk01 has been blocked for 49250 seconds. This will probably stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.
10-27-2017 09:07:22.168 +0900 INFO TcpOutputProc - Removing quarantine from idx=xxx.xxx.xxx.xxx:9997
10-27-2017 09:07:22.199 +0900 INFO TcpOutputProc - Connected to idx=xxx.xxx.xxx.xxx:9997, pset=0, reuse=0.
10-27-2017 09:07:22.714 +0900 INFO TailReader - ...continuing.
10-27-2017 09:07:22.885 +0900 INFO LMStackMgr - should rollover=true because _lastRolloverTime=1508943600 lastRolloverDay=1508943600 snappedNow=1509030000
10-27-2017 09:07:22.901 +0900 INFO LMStackMgr - finished rollover, new lastRolloverTime=1509062842

it seems like HF did not read the new log file which it should.
after i reboot the HF splunkd, it started to send all logs again.

does anyone have any idea for the work-around other than rebooting HF's splunkd?

Re: Heavy forwarder not sending logs (Windows)

koshyk — Fri, 27 Oct 2017 12:02:36 GMT

are you connected to your Indexers directly or using indexerDiscovery?

Re: Heavy forwarder not sending logs (Windows)

peterchenadded — Tue, 29 Sep 2020 16:30:12 GMT

Did you try reloading the inputs?

./splunk _internal call /services/data/inputs/monitor/_reload -auth admin:changeme

It might help.

Re: Heavy forwarder not sending logs (Windows)

hkizuka — Mon, 30 Oct 2017 05:14:01 GMT

looking at the indexer directly.

Re: Heavy forwarder not sending logs (Windows)

hkizuka — Mon, 30 Oct 2017 05:14:46 GMT

thanks! i'll try when it happens again!

topic Re: Heavy forwarder not sending logs (Windows) in Getting Data In

Heavy forwarder not sending logs (Windows)

Re: Heavy forwarder not sending logs (Windows)

Re: Heavy forwarder not sending logs (Windows)

Re: Heavy forwarder not sending logs (Windows)

Re: Heavy forwarder not sending logs (Windows)