Is it possible to route data to idle indexers ?

[splunk_force_as]

Hello,

Recently, we've been experiencing full typing queues (blocked queues) in our Splunk deployment. As a result, the forwarders will stop sending data to the indexing layer until the size of the queues drop below a certain size. Two solutions are minimizing our usage of index time configurations and increase our processing power by adding more indexers, which is our plan. The only issue I have with increasing our processing power is that i'm not 100% convinced that adding more indexers will help because I've noticed (via DMC) that we will have full typing queues on some of our indexers, meanwhile on other indexers, all queues are empty (idle) but Splunk will still stop sending data to the indexing layer(Queues blocked for more than XXX seconds) . Is this some sort of bug? We are using Splunk 6.3.2. Is it possible to configure universal forwarders to re-route data to another indexer if the initial indexer has full queues? ( Currently, we are using and enforcing autoload balancing every 30 secs). OR will I need to implement a custom mechanism? Would converting our intermediate UF forwarders to heavy forwarders be a better option for processing power? And using these heavy forwarders to perform parsing and merging and have the indexers perform typing and indexing?

[sowings]

The forwarders can't switch to an (idle) indexer until they reach EOF on the file they're currently reading. A forwarder can typically overwhelm a single indexer with straight up throughput, so it won't be able to reach EOF on busy files because it's being throttled by the indexer. If your events are small (say, < 10kB) you can use the forceTimeBasedAutoLB option on the forwarders to allow it to switch before EOF (magic, and blog posts).

Also, I'll point out that a full agg queue is related to TRANSFORMS and/or SEDCMD. If you can reduce the number of these acting on the data, that could help your overall throughput. If you need them, consider SHOULD_LINEMERGE as an alternative optimization. The details of the implementation of this will streamline certain parts of the indexing process. See here for more information.