Expanding my splunk architecture

[haleyh44]

Currently, I have a single Splunk server that is performing all the necessary functions. However, I would like to expand my infrastructure by deploying two new physical servers: one for an additional indexer and another for a dedicated search head. I am using Windows Server 2019.I would appreciate guidance on the best approach to achieve this. Specifically, I would like to know the steps involved in setting up another indexer and search head.

Any advice or guidance is appreciated! 

[gcusello]

Hi @haleyh44 ,

one additional information: do you want HA on your data or not?

to have HA you need to create an Indexer Cluster, that requires an additiona machine (Cluster Manager) that cannot be one of the others.

Anyway, the two new machines have different requirements, in terms of Disk Space: the new Indexers should have the same storage of the old server.

If you don't want HA, you have to:

• install Splunk on the two new servers,
• copy the indexes.conf and the Technology Add-Ons from the old server to one of the other two that will be one of the two Indexers,
• copy all the apps from the old server to the new server that will be the Search Head,
• configure the Search head for a Distributed Search as described in the links shared by @JohnEGones 

If you want HA, you have to:

• install Splunk on three new servers,
• Configure an Indexers Cluster on the old server and two of the new ones,
• copy the indexes.conf and the Technology Add-Ons from the old server to the Cluster Manager,
• copy all the apps from the old server to the new server that will be the Search Head,
• configure the Search head for a Distributed Search as described in the links shared by @JohnEGones 

for more infos about Splunk architectures see at https://www.splunk.com/en_us/pdfs/tech-brief/splunk-validated-architectures.pdf

Ciao.

Giuseppe

[haleyh44]

Hey Giuseppe,

I followed all of your steps for HA configuration. Built a new Search head, master node, and new indexer.

I enabled clustering and added the search head and newly built indexer to the cluster. Once i added the once standalone splunk server to the cluster the splunk service wouldnt start.

And now if fails when i try starting it. Any idea on why this would be?

[isoutamo]

Have you read and followed the steps in this document https://docs.splunk.com/Documentation/Splunk/9.3.0/Indexer/Migratenon-clusteredindexerstoaclusterede... ?

Based on your comments and questions and where you currently are, I doubt it! 

We cannot help you without knowing what you have exactly done! I hope that you have write down journal and you could share it. 

Also you must check what you have on nodes’ splunkd.log.

[haleyh44]

Steps Taken:

1. Installed Splunk Enterprise on all new servers.
2. Enabled clustering on the designated manager node.
3. Configured clustering on the new indexer, adding it as a peer node.
4. Enabled clustering and added the new server as a search head.
After verifying that the newly added servers appeared on the manager node, I attempted to enable clustering on the existing standalone Splunk server and add it as a peer node. However, when I tried to restart the Splunk services, they wouldn't start. I had to remove the clustering stanza for the services to start successfully. I'm unsure where I went wrong or if I missed a step, but it seems that adding the standalone server to the newly created cluster prevents it from starting unless I remove the clustering stanza.

[isoutamo]

Have all those servers exactly same splunk version?

You said that you also add SH to this cluster. What you are actually meaning with this?

What you are finding old indexer’s splunkd.log after you try to add it as a cluster peer? And how you are adding it into cluster (cli, edit config files)?

[haleyh44]

All of this has been through the GUI.

On the search head I enabled clustering and added it as the Search peer. Is that not the way to do it?

[PickleRick]

Don't take it wrong but please leave this to someone who has the skills and experience.

It's not something that cannot be done but you haven't even tried to test it in a lab environment but you're trying to change your production env based on bits and pieces of advice you're getting on an internet forum.

I suppose your already indexed data and work already done on your infrastructure is worth more than the bucks you're trying to save by either using external help (there are friendly Splunk Partners in every region) or at least investing in your own abilities by taking a course or two, digging through te docs, building and breaking a few lab environments.

While this is not something that's difficult for a seasoned Splunk admin, there are several things that can go wrong and you wouldn't want to lose your data because of misconfiguring your indexes or something like that.

And don't do multiple changes at the same time. Separating a search-head from existing AIO setup is one thing. Clustering indexers is another. Don't do too many things in one step.

[haleyh44]

Can i keep my once standalone server now one of my indexers as the deployment server? Or do i need to designate another server as the deployment server?

[haleyh44]

Currently working on these steps.

I have copied the indexes.conf from the standalone to the master node. Will i need to copy that config file to the new indexer as well?

[haleyh44]

Hey Giuseppe

Thanks for your response this is exactly what i was looking for. Can a virtual machine suffice for an cluster manager?

I was wanting to cluster my environment for HA. If I enable clustering do I still need to copy all of my data from the original indexer and copy onto my new indexer?

Or if i enable clustering what do i need to do to replicate my data from my original indexer to my new indexer?

Another question, if I am deploying a new indexer should I update all my forwarders to send to both indexers or should i leave it to where its sending its data to the original indexer?

[isoutamo]

Hi

you can use virtual server as a CM. Just allocate enough CPU + Mem for it. There are still some part of splunk CM code which are single thread restrictions. For that reason it's more important to have enough fast cpu and also enough memory to run it. Also you should keep care that you don't allocate two much resources for VM vs. what you have in your real virtualization host. Also over allocation mem or cpu is not good for splunk.

Also all your indexers must (should) have identical, otherwise there will be some issues later or you will not use their all resources.

When you take indexer cluster into use you can also take indexer discovery. This handling what are indexers where to send events from UFs.

I propose that you look at least the next docs:

• https://docs.splunk.com/Documentation/Splunk/latest/Indexer/Aboutclusters
• https://lantern.splunk.com/Splunk_Success_Framework/Platform_Management/Indexing_and_search_architec...
• https://docs.splunk.com/Documentation/SVA/current/Architectures/About?

Personally I prefer linux over Windows for splunk especially when you have more than one splunk server.

Also if you have any doubt that you need scale this over two site then you could/should create multisite cluster into one site. This will be easier to expand another site if/when you need that kind of HA / disaster recovery capabilities.

r. Ismo

[gcusello]

Hi @haleyh44 ,

Yes, you can use a VM as Cluster Manager, you should give it the minimal CPU and RAM requirements and it's preferable if you could give them, if you haven't you could also try with less configuration (8CPUs and 8 GB RAM).

old data cannot be replicated between Indexers, even if you have a cluster, only new data.

If you want to have two copies of the old data, you must manually copy them in both the Indexers, in a different not replicated index.

For new data, remember that you have to add to each stanza of your indexes.conf the option 

repFactor = auto

otherwise indexes aren't replicated

You must update all your Forwarders to send data in autoLoadBalancing to all your Indexers, you also could configure indexers_discovery (https://docs.splunk.com/Documentation/Splunk/9.3.0/Indexer/indexerdiscovery)

Ciao.

Giuseppe

[haleyh44]

Thanks!

Do i need to create and designate my cluster manager first and then cluster my indexers?

I am trying to figure out what i need to do first after making a new indexer and how i can cluster my two indexers. Also, same with my new search head. I dont know what step to take first? 

[gcusello]

Hi @haleyh44 ,

here you can find the process of migration from a stand alone indexer to a clustered environment.

https://docs.splunk.com/Documentation/Splunk/9.3.0/Indexer/Migratenon-clusteredindexerstoaclusterede...

Ciao.

Giuseppe

[JohnEGones]

Hi haleyh44,

You might consult the following:

https://docs.splunk.com/Documentation/Splunk/9.3.0/Deploy/Distributedoverview

https://docs.splunk.com/Documentation/SVA/current/Architectures/About?301=/pdfs/technical-briefs/spl...