Replication, Cluster and Licensing

[SharedSys]

We currently have splunk 4 running in two servers(None Clustered and second one is to bring up when needed). We are looking at upgrading to version 5 but at the same time we are looking at Replication and cluster too.. before we go to management and ask for resources we would like to find out answer to following question and will appreciate if some one could guide us through..

1) Can we have one way replication between peers(one of the peer going to be just a stand by).
2) Can we have search head, cluster master and indexer in same server?
3) if above roles can not be in same server, what do you think of having search head and cluster master being virtual? we currently have 8gb of data collection and at this search and reporting is not intensive.
4) if we can not have one way replication than we are looking at license for second server too am i correct?

We are thinking of setting of alias for Collection. In case of disaster change the pointer to surviving server.

we also don't have budget for another license so we have to work with what we have at this stage.

Thank you

[datasearchninja]

High Availability at index time is achieved by specifying more than indexer in outputs.conf and Splunk automatically load balances between them. Whichever indexer receives the data becomes the primary node for that data for replication purposes.

Clustering brings high availability at search time by storing multiple copies of the index data.

The simplest configuration would be to have 2 indexers that are both configured in outputs.conf for receiving data, and are both in a cluster replicating to each other. This achieve HA on both indexing and search.

It is not possible to easily configure both HA at the forwarder layer AND have some of the indexers as replication only targets in a cluster. Why do you want to do this? Is the replication target lower spec hardware?

The hack method of achieving this would be with 4 nodes and 2 clusters. Load balance the forwarders across node A of each of the 2 clusters, and then the other 2 nodes would only be replication targets.

[datasearchninja]

If you could accept my answer, that would be great!

[SharedSys]

Thanks Colin

Really Appreciated for the Answer.

Thank you

[datasearchninja]

If you do send more data than you are licensed for then you will breach your licence limit. In this case indexing will always continue. If you breach the licence more than 5 times in a 30 day period, search functionality will be disabled.

[datasearchninja]

There is no licence impact to using replication. If you currently index 8GB of your 10GB, and configure replication you will still be under the licence limit.

In regards to your output question, either configuration is possible, though in this case if you send the event to both indexers you WILL double your licence usage.

The normal config will have 2 servers in the same tcpout group, and the forwarder will alternate between them only sending each each event to one of the 2 indexers.

[SharedSys]

Hi Colin

Question again

1)If we specify two indexer in output file, Does the same event get forwarded to both indexer (when both available) or only forwarded to one indexer which ever is available first?
2)In regards to license limitation, say we have 5GB license can we keep sending 10GB of data to indexer? when this happen we know, we will not be able to search and report but will splunk keep indexing incoming data?

Thank you Again

[SharedSys]

Hi Colin

thank you Again

Main reason for one way replication is of License. We currently have 10GB license and around 8GB is used daily. At this stage we do not have budget to get another 10GB of license.

We were planning to have one indexer name in outputs.conf file. We will have this name as alias to primary indexer and when primary fail this alias is assigned to secondary indexer.

Both Indexers are pretty good server so not a issue there just the licensing issue.

Thank you Again

[datasearchninja]

1) The indexer where an event is first written to a bucket is the primary node for that bucket. Other nodes in the cluster would be replicas. One way replication could be achieved by only forwarding data to some of the indexer nodes in the cluster, the others would be replicas only.

2) The cluster master and indexes need to be different servers. The cluster master could function as a search head, but this is not recommended.

3) For both virtual is probably ok. For a search head the limit tends to be cpu, so so long as you can provide enough cpu for the number of searches run it should work. Arguably the cluster master might be better as virtual, as this is a single point of failure, and virtual machines tend to be able to made more available.

4) There is no licencing impact to the number of replicas made. Only the first index event is a licence event. Even storing 3 or 4 replicas of the data works this way. Extra licensing is only required if you are cloning the data during the forwarding phase of indexing.

[mahamed_splunk]

Now we forwarded events to single server and when primary indexer failed how does the replica copy of index is made primary?

The cluster master receives the heart beats from indexers. As soon as the heart_beat_time_outinterval is passed, then the master declares the indexer is dead and reassigns the primary to the secondary indexer.

[SharedSys]

Hi Colin

Thank you for replying so quickly

Now we forwarded events to single server and when primary indexer failed how does the replica copy of index is made primary? When events start coming to this secondary server will these events be written on different index file/database or will it be written on to index/database that was replica of other server?

How does this thing work?

Thank you again