topic Expanding Splunk installation from a single indexer to a multi-indexer + dedicated search head environment.. in Getting Data In

Expanding Splunk installation from a single indexer to a multi-indexer + dedicated search head environment..

Brian_Osburn — Wed, 08 Sep 2010 02:00:03 GMT

We're expanding our Splunk environment from a single indexer machine that does everything, to an environment that has dual indexers + a dedicated search head.

From my research, there seems to be a large amount of information on how to set things up BEFORE a system goes live. In our case, we've been live for approximately 4 months, and have a very active system.

I'd like to move a sub-set of our existing indexes to the new indexer in order to segregate the search load between the two instances (ie, moving all my weblogic indexing to the second server in order not to impact people searching windows logs).

There's a couple of questions I can't seem to find the answer to:

How do I handle the moving of the indexes from indexer1 to indexer2? I found this link http://www.splunk.com/wiki/Community:MoveIndexes but it hasn't been updated since Nov 2009, so I'm not even sure if this is the right process.
Do I even need to move my indexes? For example, say I update my forwarders to send to indexer2. Would the search head know where to send the data depending on date?

I want to get some advice before I move forward with this - especially since I'm planning on doing this next week.

Thanks! Brian

Re: Expanding Splunk installation from a single indexer to a multi-indexer + dedicated search head environment..

gkanapathy — Wed, 08 Sep 2010 05:50:59 GMT

You don't have to move the indexes at all. If you start forwarding the new data, it will be split between the two. However, querying older data will not benefit at all from the new hardware, so it will be as slow as it is now.

But if you have two indexers I would recommended as evenly and randomly distributing the data between the two, i.e., take every forwarder and make the autoLB between the two, and take every existing index and split it out between the two. It's not that hard and you get benefits of increased performance over the old data.

To split an index from a single node to two nodes:

Configure the same indexes on the new node.
Stop Splunk on both nodes.
Copy every other bucket from the old node to the new node, e.g., leave the odd-numbered buckets on the old node, move the even-numbered ones to the new node in the corresponding location.
Start Splunk back up.

Of course, if you are adding two more modes (or three more) node, you take every third (or fourth) bucket.

Re: Expanding Splunk installation from a single indexer to a multi-indexer + dedicated search head environment..

Brian_Osburn — Wed, 08 Sep 2010 08:45:54 GMT

That sounds like a plan. I'll have to do some research on the load balancing and the splitting off the buckets. You don't happen to have an automated way to do this? I have almost a terrabyte of data..

Re: Expanding Splunk installation from a single indexer to a multi-indexer + dedicated search head environment..

gkanapathy — Wed, 08 Sep 2010 10:03:44 GMT

Well, it's just copying folders, so you could just do: cp -R db_*[02468] /target/directory/ to get all the even-numbered ones, for instance.

Re: Expanding Splunk installation from a single indexer to a multi-indexer + dedicated search head environment..

Brian_Osburn — Wed, 08 Sep 2010 22:06:10 GMT

Awesome, thanks!

Re: Expanding Splunk installation from a single indexer to a multi-indexer + dedicated search head environment..

piebob — Wed, 08 Sep 2010 22:15:02 GMT

re: 1. there is a validated process documented here: http://www.splunk.com/base/Documentation/latest/Admin/Moveanindex
:)