Getting Data In
Highlighted

Management Console - Indexing Performance shows Queue Fill Ratio's are at 100% (almost)

Contributor

We have a multi-site cluster and I started noticing in DMC that some of the Queue Fill Ratio's are almost at 100%. What does that mean?

Here is a snapshot from 5 mins ago -

alt text

Each row here indicates an indexer (hidden for privacy). And I am noticing that the indexer keeps changing and one or the other is at near 100%.

We are using HTTP Event Collector to post data into Splunk and we are seeing "Server is busy" error while posting the events.

Please advice.

Highlighted

Re: Management Console - Indexing Performance shows Queue Fill Ratio's are at 100% (almost)

Splunk Employee
Splunk Employee

How many indexers are in your cluster?
What are your indexer specs (cores, memory)?
What are you using for HOT/WARM storage (kinds and number of disks, RAID level, size)?
What is your daily indexing volume?
Are you sending directly to the HTTP event collector (HEC) input on the indexers via a load balancer or do you have a Heavy Forwarder as the HEC endpoint?

Generally speaking, this is an indication that you are trying to process more load on an indexer than it can handle as indicated by your indexing queue backing up.

Any data coming into an indexer gets processed via multiple pipelines (containing one or more processors). Each of these pipelines has an input and output queue and does a specific task:

  • parsing pipeline/queue: UTF-8 conversion, line breaking, header extraction
  • merging pipeline/agg queue: line merging (multi-line events)
  • typing pipeline/queue: RegEx replacements, annotation (punct field)
  • indexing pipeline/queue: license metering, writing to disk (or syslog/TCP out [rarely])

If one of the pipelines can't keep up, it's input queue will grow as new data comes in. This effect "bubbles up" the pipeline chain, ultimately all the way back to the forwarder's output queue.

Since HEC is served via HTTP POST, there is no output queue on the sender side and the sender will get a "server busy" response if the receiver cannot accept new data.

My best guess is that you will need to add additional indexers to handle the ingest load you are trying to process.