Getting Data In

Heavy forwarder not sending logs (Windows)

hkizuka
Explorer

I've got an issue with HF not sending the logs to indexer.
Does anyone have experience with something like this?

HF was sending the log to indexer as it should until yesterday.
at one moment, indexer OS somehow got shutdown and HF didn't send any logs including internal logs even after the indexer was booted and connection was established.

HF:Windows Server 2012
indexer:Windows Server 2016
Splunk version : 6.6.3

when I checked splunkd.log in HF, I saw logs written as below


10-27-2017 09:07:18.938 +0900 WARN TcpOutputProc - Tcpout Processor: The TCP output processor has paused the data flow. Forwarding to output group splunk01 has been blocked for 49250 seconds. This will probably stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.
10-27-2017 09:07:22.168 +0900 INFO TcpOutputProc - Removing quarantine from idx=xxx.xxx.xxx.xxx:9997
10-27-2017 09:07:22.199 +0900 INFO TcpOutputProc - Connected to idx=xxx.xxx.xxx.xxx:9997, pset=0, reuse=0.
10-27-2017 09:07:22.714 +0900 INFO TailReader - ...continuing.
10-27-2017 09:07:22.885 +0900 INFO LMStackMgr - should rollover=true because _lastRolloverTime=1508943600 lastRolloverDay=1508943600 snappedNow=1509030000
10-27-2017 09:07:22.901 +0900 INFO LMStackMgr - finished rollover, new lastRolloverTime=1509062842


it seems like HF did not read the new log file which it should.
after i reboot the HF splunkd, it started to send all logs again.

does anyone have any idea for the work-around other than rebooting HF's splunkd?

0 Karma

peterchenadded
Path Finder

Did you try reloading the inputs?

./splunk _internal call /services/data/inputs/monitor/_reload -auth admin:changeme

It might help.

0 Karma

hkizuka
Explorer

thanks! i'll try when it happens again!

0 Karma

koshyk
Super Champion

are you connected to your Indexers directly or using indexerDiscovery?

0 Karma

hkizuka
Explorer

looking at the indexer directly.

0 Karma
Get Updates on the Splunk Community!

The Splunk Success Framework: Your Guide to Successful Splunk Implementations

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...

Splunk Training for All: Meet Aspiring Cybersecurity Analyst, Marc Alicea

Splunk Education believes in the value of training and certification in today’s rapidly-changing data-driven ...

Investigate Security and Threat Detection with VirusTotal and Splunk Integration

As security threats and their complexities surge, security analysts deal with increased challenges and ...