topic Re: Best Practices for Configuring dev, prod environments in Splunk in Splunk Dev

Best Practices for Configuring dev, prod environments in Splunk

caremore — Wed, 27 Feb 2019 20:04:41 GMT

Hello,
Can you please help me in understanding the best practices to design and implement the Splunk ecosystem in our organization

We have around 300 applications deployed onto Dev, Qa, Stage and Prod environments,
we have one Splunk Enterprise Licensed Stand Alone server and 10 applications's that aggregate logs to Splunk.

Current settings, and usage:
configured pool size max size of the index is 500GB
the daily limit of volume pool can consume: 11,264 MB
currently we are consuming 1-5MB

We want to have 2 Splunk systems to be created
1. for log aggregation for Dev, Qa, Stage
2. For Prod
We use Splunk for Log aggregation, Alerting, Reporting, and dashboards

So I have a few basic questions like:
what are the best practices for configuring this kind of environment considering we have 4 servers available?
1. Can License master, Deployment server, search head hosted on a single server and Indexer on another server? and use Universal forwarders redirect logs to Indexer?
2. Currently, all the logs /data is getting aggregated to Standalone Server, how can I move the dev data to Dev Splunk server once I have both Splunk Instances up and running?
3. Links/references to How to maintain Splunk Dashboards as Code in Git?
4. Links/references to Ansible Playbooks to install/Configure Splunk Universal Forwarders on the Clients.
Thanks in Advance.

Re: Best Practices for Configuring dev, prod environments in Splunk

ddrillic — Wed, 27 Feb 2019 22:29:11 GMT

You have four servers and four environments, so I would go for a standalone implementation on each server for each environment.

Re: Best Practices for Configuring dev, prod environments in Splunk

caremore — Wed, 27 Feb 2019 23:24:51 GMT

You mean, indexer, search head, deployment server - all components on stand-alone server for each environment?

Re: Best Practices for Configuring dev, prod environments in Splunk

ddrillic — Thu, 28 Feb 2019 01:44:40 GMT

Pretty much @caremore - Splunk standalone server means a single Splunk server in which all the functions - indexer, search head, deployment server, etc, are in a single instance of Splunk.

In your case, each one of these four physical servers, would host a Splunk standalone server.

Re: Best Practices for Configuring dev, prod environments in Splunk

woodcock — Thu, 28 Feb 2019 06:12:10 GMT

Q1: Can License master, Deployment server, search head hosted on a single server and Indexer on another server? and use Universal forwarders redirect logs to Indexer?
A1: This is not a supported or advisable configuration. See here:
https://docs.splunk.com/Documentation/Splunk/latest/Indexer/Systemrequirements#Additional_roles_for_the_master_node
http://docs.splunk.com/Documentation/Splunk/latest/Admin/ConfiguretheMonitoringConsole
https://answers.splunk.com/answers/380825/possible-combinations-of-splunk-instances-with-dif.htmlhttps://answers.splunk.com/answers/150606/should-we-have-splunk-deployment-server-and-cluster-master-on-the-same-instance-recommendations-please.html
https://answers.splunk.com/answers/96197/any-know-issues-with-deployment-server-and-master-on-same-machine.html
https://answers.splunk.com/answers/302606/what-is-the-best-way-to-combine-a-license-master-d.html
I often combine these together:
License master + Monitoring console + Search Head Cluster Deployer

Q2: Currently, all the logs /data is getting aggregated to Standalone Server, how can I move the dev data to Dev Splunk server once I have both Splunk Instances up and running?
A2: The only practical way to separate data once it is indexed is index-by-index and you just copy the entire directory structure where you would like it to live (dev vs. prod).

Q3: Links/references to How to maintain Splunk Dashboards as Code in Git?
A3: See here for ideas:
https://www.slideshare.net/HarryMcLaren/spldevops-making-splunk-development-a-breeze-with-a-deep-dive-on-devops-containerization-version-control-and-automation

Q4: Links/references to Ansible Playbooks to install/Configure Splunk Universal Forwarders on the Clients.
Thanks in Advance.
A4: I have not done this but it looks like plenty of people have:
https://www.google.com/search?q=ansible+splunk&rlz=1C1GCEV_en&oq=ansible+splunk&aqs=chrome..69i57j0j69i65l2j0l2.2231j1j7&sourceid=chrome&ie=UTF-8

Re: Best Practices for Configuring dev, prod environments in Splunk

sloshburch — Fri, 15 Mar 2019 20:59:24 GMT

Building on this solid answer, I will point out that the nonprod data is still production for someone's job function. I would encourage you to challenge the separation and instead consider having one Splunk environment for all of the prod and nonprod. You can separate the data itself with indexes.

This will allow comparisons of data and patterns across the environments that are the bedrock Splunk's value.

You're welcome to share back any ideas you felt separation was appropriate. Maybe you notice something I didn't consider OR maybe you will learn cool product features you didn't know.

Remember that a lab is not the same as non-prod. See Lab environment best practices for a Splunk deployment