Heavy forwarder not sending logs (Windows)

[hkizuka]

I've got an issue with HF not sending the logs to indexer.
Does anyone have experience with something like this?

HF was sending the log to indexer as it should until yesterday.
at one moment, indexer OS somehow got shutdown and HF didn't send any logs including internal logs even after the indexer was booted and connection was established.

HF:Windows Server 2012
indexer:Windows Server 2016
Splunk version : 6.6.3

when I checked splunkd.log in HF, I saw logs written as below

---

10-27-2017 09:07:18.938 +0900 WARN TcpOutputProc - Tcpout Processor: The TCP output processor has paused the data flow. Forwarding to output group splunk01 has been blocked for 49250 seconds. This will probably stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.
10-27-2017 09:07:22.168 +0900 INFO TcpOutputProc - Removing quarantine from idx=xxx.xxx.xxx.xxx:9997
10-27-2017 09:07:22.199 +0900 INFO TcpOutputProc - Connected to idx=xxx.xxx.xxx.xxx:9997, pset=0, reuse=0.
10-27-2017 09:07:22.714 +0900 INFO TailReader - ...continuing.
10-27-2017 09:07:22.885 +0900 INFO LMStackMgr - should rollover=true because _lastRolloverTime=1508943600 lastRolloverDay=1508943600 snappedNow=1509030000
10-27-2017 09:07:22.901 +0900 INFO LMStackMgr - finished rollover, new lastRolloverTime=1509062842

---

it seems like HF did not read the new log file which it should.
after i reboot the HF splunkd, it started to send all logs again.

does anyone have any idea for the work-around other than rebooting HF's splunkd?

[peterchenadded]

Did you try reloading the inputs?

./splunk _internal call /services/data/inputs/monitor/_reload -auth admin:changeme

It might help.

[hkizuka]

thanks! i'll try when it happens again!

[koshyk]

are you connected to your Indexers directly or using indexerDiscovery?

[hkizuka]

looking at the indexer directly.