Best Practices for Configuring dev, prod environments in Splunk

[caremore]

Hello,
Can you please help me in understanding the best practices to design and implement the Splunk ecosystem in our organization

We have around 300 applications deployed onto Dev, Qa, Stage and Prod environments,
we have one Splunk Enterprise Licensed Stand Alone server and 10 applications's that aggregate logs to Splunk.

Current settings, and usage:
configured pool size max size of the index is 500GB
the daily limit of volume pool can consume: 11,264 MB
currently we are consuming 1-5MB

We want to have 2 Splunk systems to be created
1. for log aggregation for Dev, Qa, Stage
2. For Prod
We use Splunk for Log aggregation, Alerting, Reporting, and dashboards

So I have a few basic questions like:
what are the best practices for configuring this kind of environment considering we have 4 servers available?
1. Can License master, Deployment server, search head hosted on a single server and Indexer on another server? and use Universal forwarders redirect logs to Indexer?
2. Currently, all the logs /data is getting aggregated to Standalone Server, how can I move the dev data to Dev Splunk server once I have both Splunk Instances up and running?
3. Links/references to How to maintain Splunk Dashboards as Code in Git?
4. Links/references to Ansible Playbooks to install/Configure Splunk Universal Forwarders on the Clients.
Thanks in Advance.

[woodcock]

Q1: Can License master, Deployment server, search head hosted on a single server and Indexer on another server? and use Universal forwarders redirect logs to Indexer?
A1: This is not a supported or advisable configuration. See here:
https://docs.splunk.com/Documentation/Splunk/latest/Indexer/Systemrequirements#Additional_roles_for_...
http://docs.splunk.com/Documentation/Splunk/latest/Admin/ConfiguretheMonitoringConsole
https://answers.splunk.com/answers/380825/possible-combinations-of-splunk-instances-with-dif.htmlhtt...
https://answers.splunk.com/answers/96197/any-know-issues-with-deployment-server-and-master-on-same-m...
https://answers.splunk.com/answers/302606/what-is-the-best-way-to-combine-a-license-master-d.html
I often combine these together:
License master + Monitoring console + Search Head Cluster Deployer

Q2: Currently, all the logs /data is getting aggregated to Standalone Server, how can I move the dev data to Dev Splunk server once I have both Splunk Instances up and running?
A2: The only practical way to separate data once it is indexed is index-by-index and you just copy the entire directory structure where you would like it to live (dev vs. prod).

Q3: Links/references to How to maintain Splunk Dashboards as Code in Git?
A3: See here for ideas:
https://www.slideshare.net/HarryMcLaren/spldevops-making-splunk-development-a-breeze-with-a-deep-div...

Q4: Links/references to Ansible Playbooks to install/Configure Splunk Universal Forwarders on the Clients.
Thanks in Advance.
A4: I have not done this but it looks like plenty of people have:
https://www.google.com/search?q=ansible+splunk&rlz=1C1GCEV_en&oq=ansible+splunk&aqs=chrome..69i57j0j...

[sloshburch]

Building on this solid answer, I will point out that the nonprod data is still production for someone's job function. I would encourage you to challenge the separation and instead consider having one Splunk environment for all of the prod and nonprod. You can separate the data itself with indexes.

This will allow comparisons of data and patterns across the environments that are the bedrock Splunk's value.

You're welcome to share back any ideas you felt separation was appropriate. Maybe you notice something I didn't consider OR maybe you will learn cool product features you didn't know.

Remember that a lab is not the same as non-prod. See Lab environment best practices for a Splunk deployment

[ddrillic]

You have four servers and four environments, so I would go for a standalone implementation on each server for each environment.

[caremore]

You mean, indexer, search head, deployment server - all components on stand-alone server for each environment?

[ddrillic]

Pretty much @caremore - Splunk standalone server means a single Splunk server in which all the functions - indexer, search head, deployment server, etc, are in a single instance of Splunk.

In your case, each one of these four physical servers, would host a Splunk standalone server.