Why did my heavy forwarder stopped forwarding data abruptly?

[neelshah]

The HF in our environment stops forwarding data abruptly, while the Splunk process is still running. Initial investigation shows no issues with HF. On Restarting the HF it enables the data forwarding again before abruptly stops again with no clues in splunkd.log. There is no specific time no network issues.

Anyone with similar issues? any solutions?

[FrankVl]

How exactly do you come to the conclusion that your HF has stopped forwarding data?

[neelshah]

Like anyone else, data from the particular host is more than 24hrs late. After restart, all that missing data appears and updated to latest time.

[ranjitbrhm1]

Im pretty sure this is not exactly the solution but try turning on the "Store a local copy" for lets say 5 minutes to verify that data is actually injecting into your heavy forwarder. then also login to the splunk console and then try telnetting into the receiving indexer's receiver port to make sure its not a network issue. Normally if there is some errors, there used to be a popup complaining that the receiver downstream is not receiving data and the data flow is stopped or something down the line like that. If both of the things are working fine then it might be something else.

[neelshah]

The Web page for forwarder is disabled. hence no means to get that popup. I'll try your suggestion of store local copy

[teunlaan]

Are you sure it stopped forwarding data?
If it is processing large files, it can could be running in "batchmode". What means it will try to send out the complete file before switching to the next.

And how many parallelIngestionPiplines are you runnig? It the system in in batchmode, nothing else will come though. When adding an extra pipleline, the "extra pipeline" can process the other data (will need more CPU prower)

[neelshah]

parallelIngestionPiplines =3

I understand batch mode, but then a couple of times I have waited for 1-2 days for data to arrive and data only starts arriving after restart