Why is heavy forwarder not sending metrics data?

[MaorMagori]

Hello 👋

I'm having trouble with metric data not being sent from our HF to our Enterprise deployment.
I'll add a diagram later to better explain myself. For now, our deployment looks a bit like this:

Monitored File -----> UF -----> HF ------> Indexer (WORKS!)

We decided to send our collectd metric data through the UF but for some reason, the metric data got lost while the monitored file data reached our indexer:

Monitored File ------┐

                                        ├------> UF --------> HF ------> Indexer (Only file works)

Metric Data ---------┘

As part of the debugging test I ran to solve this I realized that sending the metric data straight to the indexer while having the file data pass through the HF works! while assuring me that the problem is not with the UF or the metric data, I still need the HF to act as a proxy in our network.

Since we have the Forwarder License on the HF we can't run searches on it and `splunkd.log` or `metrics.log` are not showing any errors.
Can anyone point me to some setting on the HF that I might miss? I've been trying to solve this for a couple of days but can't seem to make any progress.

Edit: Let me know what .conf files or other configurations you need me to share.

[isoutamo]

What splunk versions your UFs, HFs and other nodes are? There was some incompatibilities between pre 8.x vs. the newer in metrics data.

[MaorMagori]

Thanks for the reply!

UF and HF version is 8.1.2

All of the rest are 8.1.9

Could that be it?

[isoutamo]

Are you getting any metrics data from those HFs? And is it the same situation with all HFs or only some e.g. which are working as gateway/intermediate HF.