Heavy Forwarder queue saturation for sending logs to Universal Forwarders

[jeremieQuiviger]

Hello, 

I recently had to deploy a Heavy Forwarder in my infrastructure in order to perform transformations using a custom app. The current flow is as follows :

UF -> Heavy Forwarder relay -> Universal Forwarder relay -> Indexers

I am now observing a queue full issue on all Heavy Forwarders, while the queues on the next Universal Forwarders relay remain free.

I am therefore questioning the suitability of this architecture. I would like to know whether the Heavy Forwarder is still supposed to be at the end of the log flow, sending data directly to the Indexers, or if it can be positioned upstream of a Universal Forwarder that relays the data to the indexers.

Could this intermediate layer of Universal Forwarder be the cause of the queue saturation?

Thank you in advance for your help.

[PickleRick]

Well... Technically, this setup should generally work although it is not officially supported and if my memory serves me right there can be some issues around useACK.

@gcusello 's hint about thruput limits might be the right path. The thing is the thruput processor comes _before_ the ingestion pipeline (at least that's how I read the limits.conf spec file) so your input would get throttled but still you'd have empty queues "inside" the forwarder.

[gcusello]

Hi @jeremieQuiviger ,

why are you using another UF after the HF, you can use the HF for parsing and relay.

Anyway, did you checked the throughtput through the UFs? by default they have 256 kb, instead HF is unlimited.

The best solution is to replace the UF relay with an HF.

Ciao.

Giuseppe