How many indexers are in your cluster?
What are your indexer specs (cores, memory)?
What are you using for HOT/WARM storage (kinds and number of disks, RAID level, size)?
What is your daily indexing volume?
Are you sending directly to the HTTP event collector (HEC) input on the indexers via a load balancer or do you have a Heavy Forwarder as the HEC endpoint?
Generally speaking, this is an indication that you are trying to process more load on an indexer than it can handle as indicated by your indexing queue backing up.
Any data coming into an indexer gets processed via multiple pipelines (containing one or more processors). Each of these pipelines has an input and output queue and does a specific task:
- parsing pipeline/queue: UTF-8 conversion, line breaking, header extraction
- merging pipeline/agg queue: line merging (multi-line events)
- typing pipeline/queue: RegEx replacements, annotation (punct field)
- indexing pipeline/queue: license metering, writing to disk (or syslog/TCP out [rarely])
If one of the pipelines can't keep up, it's input queue will grow as new data comes in. This effect "bubbles up" the pipeline chain, ultimately all the way back to the forwarder's output queue.
Since HEC is served via HTTP POST, there is no output queue on the sender side and the sender will get a "server busy" response if the receiver cannot accept new data.
My best guess is that you will need to add additional indexers to handle the ingest load you are trying to process.
