Pros and Cons for Multiple Splunk Indexers

[rongruspe]

Our present architecture now is single indexer, and multiple universal forwarders; However, it's getting slower when we have multiple panels and multiple search strings. Wondering if the solution is to have multiple indexers?

[esix_splunk · Accepted solution]

What you're describing is search concurrency vs indexing capability. Number of indexers wont particularly effect the search results ( I say that, but there is a causation with multiple indexers / clustering and search distribution.) This is especially true if you're running in an 'all-in-one' IDX / SH case..

Simple answers:
1) Increasing indexer numbers will decrease disk i/o and minimal cpu / memory usage on the boxes, allowing more potential search
2) Separating your SH / IDX into different boxes will also reduce overhead
3) Check your CPU and MEm utilization on your current box and see if you are actually resource constrained. CPU / Memory utilization is typically search related, where-as Disk i/o utilization will be more along the lines of actually indexing. If you're on a VM, add more memory and vcpu.
4) On the Search side, perhaps you should be looking at scheduled searches vs real-time (run search when dashboard loads) type searches. (Savedsearches and loadjob..) This typically is one way to help alleviate dashboard loadtime issues, but it is does depend on your search requirements.

[rongruspe]

Got it. Thanks for giving detailed information on this. Follow up question: will it consume bandwidth? We will be deploying boxes remotely, and current architecture is that these boxes have universal forwarders. The main Indexer is here in our office.

If we deploy indexers to these boxes, instead of universal forwarders, I would imagine, that if I run a search query from the main server, it will connect to these remote boxes all the time? Isn't it going to be slower that way? Specially when at the worst case, the network is slow, even at load balancing?

[esix_splunk]

When you run a search from a SH, it distributes the search to all peers. So if you have bandwidth issues betweent he SH and IDX, then you wont get all search results or will get timeouts from the peers. It really depends on your use case. Typically, indexers should be close to the SH. Remote sites should have aggregated HF instances that collect and hold the splunk feeds until bandwidth is available.

[rongruspe]

So this means, you would recommend having one SH, multiple IDX (both of which are deployed in the office) and remote sites all have HF instances?