Hey everyone!
We're currently in the process of getting ready to deploy a Splunk Cloud instance to migrate our local on-prem version from. Currently, our environment is a hodge-podge of installs, including completely unmanaged universal forwarders, a couple heavy forwarder clusters, and so on. We also have resources both in our local datacenter and in various cloud providers.
I've been of the thought for a while that we should toss the deployment servers into a container environment. I was curious if anyone had experience with doing this?
Here's the design I want to build towards:
Now, I know this is massively over-engineering the solution. We've got a couple thousand potential endpoints to manage, so a single standalone deployment server would do the trick. But I want to try this route for two reasons. First, I think it will scale better - especially if I get it agnostic enough that we can use it to deploy to AWS or Azure and get cloud-local deployment servers. Second, and perhaps more importantly, I want to practice and stretch my skills with containers. I've already worked with our cloud team to build out a Splunk Connect for Kubernetes setup in order to monitor our pods and Openshift environment. I want to take this opportunity to learn.
Hi @AHBrook,
I suppose that you're speaking of a private cloud and not about Splunk Cloud.
Anyway, there isn't any problem to have DS in Cloud, even if i prefer to have Deployment Server in on on-prem system, mainly if you have more systems on prem than in your private cloud!
I'd choose the solution that permits to open less firewall connections between your systems and DS.
Anyway, ansering to your questions:
In addition, you spoke about Heavy Forwarders Cluster: there isn't any HF Cluster Feature in Splunk: it's a best practice to have redundant HFs that means two or more DSs but it isn't a Cluster and load balancing is managed by Splunk so you don't need Load Balancer for DSs.
About the problem that your DS has to manage more than 2000 clients, one DS can manage your clients withot problems, eventually you could give more resources to your DS (24 CPUs instead of 12 and 24/32 GB RAM instead of 12) and eventually use more DS.
If you have more DSs, you have always to use the feature that there's one Main DS that manages its clients and the other DSs.
One final hint: if you have to manage a so large architecture, you need to engage a Splunk Architect for designing and managing it: this isn't a job for questions in Community, eventually ask to your Manager to pay for you an Splunk Architect Training, it's safer for your company!
Ciao.
Giuseppe
Hi @AHBrook,
I suppose that you're speaking of a private cloud and not about Splunk Cloud.
Anyway, there isn't any problem to have DS in Cloud, even if i prefer to have Deployment Server in on on-prem system, mainly if you have more systems on prem than in your private cloud!
I'd choose the solution that permits to open less firewall connections between your systems and DS.
Anyway, ansering to your questions:
In addition, you spoke about Heavy Forwarders Cluster: there isn't any HF Cluster Feature in Splunk: it's a best practice to have redundant HFs that means two or more DSs but it isn't a Cluster and load balancing is managed by Splunk so you don't need Load Balancer for DSs.
About the problem that your DS has to manage more than 2000 clients, one DS can manage your clients withot problems, eventually you could give more resources to your DS (24 CPUs instead of 12 and 24/32 GB RAM instead of 12) and eventually use more DS.
If you have more DSs, you have always to use the feature that there's one Main DS that manages its clients and the other DSs.
One final hint: if you have to manage a so large architecture, you need to engage a Splunk Architect for designing and managing it: this isn't a job for questions in Community, eventually ask to your Manager to pay for you an Splunk Architect Training, it's safer for your company!
Ciao.
Giuseppe
Thanks for the detailed replies!
That's a good point about the redundancy aspect with the Deployment Server. I guess it makes sense that I shouldn't have to run more than 1 at any given time, unless we go 1 cloud, 1 on-prem. I'd still like them to share a code base / setup as much as possible, just to make updating/maintenance as easy as possible.
In the environment we are building out, the indexers are search heads are with Splunk Cloud. The Deployment Server and Heavy Forwarders are going to be managed by us and talk to the Splunk Cloud environment.
I apologize for not being clear about the "shared storage." I was referring to the splunk/etc/deployment-apps folder. I recognize that sharing storage for indexers is a bad idea(tm).
And yeah, the idea with the deployment servers having a load balancer would be for the initial check in of deployed components. I haven't gotten this far yet, but I'd like to make it so that whenever a Splunk UF or HF is deployed in our environment, it knows where to look to register with our Splunk instance. I don't know yet if that is possible to have them check into our Cloud instance, or into our Deployment servers directly.
The clarification on Heavy Forwarders is well taken. 🙂 I got my terms mixed up a bit. The plan we have is for horizontal scaling, where our networking syslogs and other systems that don't have local log storage would not have to care which node they talk to.
The architecture element is also well taken. I've so far only gotten my Splunk Power User certificate, but I have taken Splunk System and Data Administraton (I need to get on taking that cert exam here once things calm down). I hope to get Architect someday. My coworker has also taken Splunk Cloud Administrator courses. We are pretty well engaged with Splunk when it comes to architecture and planning and setting up our environment, but again this is a bit of a side project / proof of concept for me. 🙂
Hi @AHBrook,
as I said, my hont is to have only one DS, at least one on-prem ona one in cloud, but in this case, you have to configure the second one as a slave of the first.
About the problem of a new Forwarder, I hint to create a dedicated App (called e.g. TA_Forwarders) that contains only three files:
In this way, when you install a new forwarder, you have only to copy this app in $SPLUNK_HOME/etc/apps and restart local Splunk, so you'll have all the correct configurations by the DS and you don't need any shared folder for configurations.
HFs orizzontal scaling is always guaranteed: you can anytime add a new HF or add more resources to the existing ones.
As I said, you have a very large architecture that requires a Splunk Architect review, eventully at the beginning an external one and then you!
See next time!
Please accept one answer for the other people of Community
Ciao and happy splunking
Giuseppe
P.S.: Karma Points are appreciated 😉