Moving from one indexer to two

[yongly]

As we index more data and get more users using our Splunk system, our single splunk deployment is getting a bit loaded so the next phase is to add another indexer to the deployment to help with this.

The current deployment:
Server1 = indexer and search head
Server2 = heavy forwarder/filter and deployment server

Target deployment:
Server1 = indexer and search head
Server2 = heavy forwarder/filter and deployment server
Server3 = indexer and search peer.

So, is it a better idea to copy index data across to the new server from server1 and set up load balancing for all indexes or split up the indexes so server1 has indexes for one business unit, and server2 has indexes for another Business unit?

Is there updated documentation anywhere for this? My search through the forums and documentation only show instructions that are a little old and not for the latest splunk version.

[kristian_kolb]

I would say that a combination of both would - at least in theory - give the best results, i.e. having the indexes split over more than one indexer will improve results, since the data will be retrieved from more than one source (=less work for each indexer involved).

Also, depending on the data stored by each business unit, and the nature of the searches made, it could be beneficial to let each BU have its own index, e.g. if BU1 will never want to (or be allowed to) search for data from BU2, there would be little point in storing them in the same index, since that would make the relvant data being stored less densely. This becomes even more true if, for example, BU1 is responsible for 90% of the indexed events, and BU2 for 10%. Then BU2 would note a significant performance increase in the searches, since in a single index scenario 90% of the events would be 'useless'.

This however also has a lot to do with whether the searches are free-text oriented or more strictly defined in terms of sourcetype, source and host restrictions. So similarly, it may be a
good idea to split different sourcetypes or hosts into separate indexes. But the answer for which solution is the best is (as always): "it depends".

Hope this helps,

Kristian

[ChrisG]

Hello, there is a current (5.0.2) version of the Distributed Deployment Manual; you might want to look at the hardware requirements chapter, particularly Distribute indexing and searching and the topics that follow. Someone else can probably give you the specific answer you are looking for (splitting the indexes or not)...I don't know myself. It probably has a lot to do with the amount of data you're looking at, though.