Archive

Cloning and shutting down an indexer

Communicator

So originally I thought the way cloning would work between two indexers was that data from a forwarder setup to clone would send identical events to each indexer - this part does work fine. However, when I bring down one indexer for a few minutes on a restart or whatnot, I thought the forwarders would send to the indexer that was up, while keeping track of what needed to be sent to the second indexer when it came back online.

It appears the indexer that was down never gets the data that it missed while it was down once it comes back online. Does cloning work in that as long as one of the indexers is online that is good enough for those events? Doesn't seem right as I would think cloned should be identical, but it appears that's what I'm seeing.

Is the only way around this to clone to two sets of autolb'ed indexers (ie. using four total indexers)? Or what I'm seeing I shouldn't be seeing?

Thanks,

Scott

Tags (1)
1 Solution

Splunk Employee
Splunk Employee

That is exactly the intended behavior. You configure your forwarders to send duplicate data to 2 indexers, and if one is down, the other can be used. In case of data lost, one can be used to refill the other, but a merge can be troublesome because you are going to have overlapping buckets. The forwarders have no way of knowing where they need to pick up in order to make the data identical. Auto-LB would ensure that in the group, all data is recieved. If you had a search head that was set up to use both indexers as search peers, this would allow you to be sure that between the two indexers, you'd have the entire set of data.

http://www.splunk.com/base/Documentation/4.2/Deploy/Clonedatad

View solution in original post

0 Karma

Splunk Employee
Splunk Employee

Regardless of what the documentation currently, using forwarder "cloning" to make a copy of your data is a bad idea and doesn't do what you would expect, precisely because of what you see. I would go so far as to say the documented recommendations are wrong.

What you should instead do is forward to your primary indexer(s), then each primary indexer would in turn forward to a matching secondary indexer, i.e., each primary would have a "clone" backup. This setup gives you what you're probably actually looking for.

If you are using a cluster of primary indexers and load balancing over them, then if one goes down, the rest of the cluster will be fine. In turn, the secondary that is connected to the primary will stop receiving data, and therefore will remain in sync.

0 Karma

Communicator

Thanks for the clarification. I meant I was going to go the cloned to two distinct lb groups, not feed from the primary downstream - but I understand what you describe as an option like that if I wanted to go that route.

0 Karma

Splunk Employee
Splunk Employee

Well, just to be clear, the secondary set is not really LB. Each secondary server is "clone" of one of the primaries (the primaries are LB by the forwarders). I think that is what I said, but I want to be sure you don't forward from the primaries to all of the secondaries, but only one each.

0 Karma

Communicator

Thanks, I think my best bet at this point is the latter of what you meantion, cluster to two lb sets.

0 Karma

Splunk Employee
Splunk Employee

That is exactly the intended behavior. You configure your forwarders to send duplicate data to 2 indexers, and if one is down, the other can be used. In case of data lost, one can be used to refill the other, but a merge can be troublesome because you are going to have overlapping buckets. The forwarders have no way of knowing where they need to pick up in order to make the data identical. Auto-LB would ensure that in the group, all data is recieved. If you had a search head that was set up to use both indexers as search peers, this would allow you to be sure that between the two indexers, you'd have the entire set of data.

http://www.splunk.com/base/Documentation/4.2/Deploy/Clonedatad

View solution in original post

0 Karma

Communicator

Thanks for the answer. I guess in my mind I think of cloned as two things having exact where autolb would be the spread amongst the members with at least one member getting a copy. All in all, I'll have to move to the combo of the two. Thanks again.

0 Karma