Heavy Forwarder queue saturation for sending logs to Universal Forwarders

[jeremieQuiviger]

Hello, 

I recently had to deploy a Heavy Forwarder in my infrastructure in order to perform transformations using a custom app. The current flow is as follows :

UF -> Heavy Forwarder relay -> Universal Forwarder relay -> Indexers

I am now observing a queue full issue on all Heavy Forwarders, while the queues on the next Universal Forwarders relay remain free.

I am therefore questioning the suitability of this architecture. I would like to know whether the Heavy Forwarder is still supposed to be at the end of the log flow, sending data directly to the Indexers, or if it can be positioned upstream of a Universal Forwarder that relays the data to the indexers.

Could this intermediate layer of Universal Forwarder be the cause of the queue saturation?

Thank you in advance for your help.

[thahir]

Hi @jeremieQuiviger 

UF relay layer is almost certainly the cause of your queue saturation, and the architecture itself is the root problem.

Heavy Forwarder output capacity is substantial — it's a full Splunk instance with configurable output queues, TCP connection pooling, and parallel pipelines.

but the UF's are designed to be sources not relays

in your case HF pushes data at full speed, the UF relay can't keep up, and backpressure propagates back to fill the HF's output queue.

as @PickleRick  said technically splunk allows HF->UF relay->Indexers but its not supported and recommended for Prodcution

it should be like UF (collect) → HF (transform) → HF relay → Indexers

[PickleRick]

Well... Technically, this setup should generally work although it is not officially supported and if my memory serves me right there can be some issues around useACK.

@gcusello 's hint about thruput limits might be the right path. The thing is the thruput processor comes _before_ the ingestion pipeline (at least that's how I read the limits.conf spec file) so your input would get throttled but still you'd have empty queues "inside" the forwarder.

[gcusello]

Hi @jeremieQuiviger ,

why are you using another UF after the HF, you can use the HF for parsing and relay.

Anyway, did you checked the throughtput through the UFs? by default they have 256 kb, instead HF is unlimited.

The best solution is to replace the UF relay with an HF.

Ciao.

Giuseppe