Getting Data In

Expanding Splunk installation from a single indexer to a multi-indexer + dedicated search head environment..

Brian_Osburn
Builder

We're expanding our Splunk environment from a single indexer machine that does everything, to an environment that has dual indexers + a dedicated search head.

From my research, there seems to be a large amount of information on how to set things up BEFORE a system goes live. In our case, we've been live for approximately 4 months, and have a very active system.

I'd like to move a sub-set of our existing indexes to the new indexer in order to segregate the search load between the two instances (ie, moving all my weblogic indexing to the second server in order not to impact people searching windows logs).

There's a couple of questions I can't seem to find the answer to:

  1. How do I handle the moving of the indexes from indexer1 to indexer2? I found this link http://www.splunk.com/wiki/Community:MoveIndexes but it hasn't been updated since Nov 2009, so I'm not even sure if this is the right process.
  2. Do I even need to move my indexes? For example, say I update my forwarders to send to indexer2. Would the search head know where to send the data depending on date?

I want to get some advice before I move forward with this - especially since I'm planning on doing this next week.

Thanks! Brian

Tags (2)
1 Solution

gkanapathy
Splunk Employee
Splunk Employee

You don't have to move the indexes at all. If you start forwarding the new data, it will be split between the two. However, querying older data will not benefit at all from the new hardware, so it will be as slow as it is now.

But if you have two indexers I would recommended as evenly and randomly distributing the data between the two, i.e., take every forwarder and make the autoLB between the two, and take every existing index and split it out between the two. It's not that hard and you get benefits of increased performance over the old data.

To split an index from a single node to two nodes:

  1. Configure the same indexes on the new node.
  2. Stop Splunk on both nodes.
  3. Copy every other bucket from the old node to the new node, e.g., leave the odd-numbered buckets on the old node, move the even-numbered ones to the new node in the corresponding location.
  4. Start Splunk back up.

Of course, if you are adding two more modes (or three more) node, you take every third (or fourth) bucket.

View solution in original post

piebob
Splunk Employee
Splunk Employee

re: 1. there is a validated process documented here: http://www.splunk.com/base/Documentation/latest/Admin/Moveanindex
:)

0 Karma

gkanapathy
Splunk Employee
Splunk Employee

You don't have to move the indexes at all. If you start forwarding the new data, it will be split between the two. However, querying older data will not benefit at all from the new hardware, so it will be as slow as it is now.

But if you have two indexers I would recommended as evenly and randomly distributing the data between the two, i.e., take every forwarder and make the autoLB between the two, and take every existing index and split it out between the two. It's not that hard and you get benefits of increased performance over the old data.

To split an index from a single node to two nodes:

  1. Configure the same indexes on the new node.
  2. Stop Splunk on both nodes.
  3. Copy every other bucket from the old node to the new node, e.g., leave the odd-numbered buckets on the old node, move the even-numbered ones to the new node in the corresponding location.
  4. Start Splunk back up.

Of course, if you are adding two more modes (or three more) node, you take every third (or fourth) bucket.

Brian_Osburn
Builder

Awesome, thanks!

0 Karma

gkanapathy
Splunk Employee
Splunk Employee

Well, it's just copying folders, so you could just do: cp -R db_*[02468] /target/directory/ to get all the even-numbered ones, for instance.

0 Karma

Brian_Osburn
Builder

That sounds like a plan. I'll have to do some research on the load balancing and the splitting off the buckets. You don't happen to have an automated way to do this? I have almost a terrabyte of data..

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...