Advice for setting up a Splunk production environment

[mrkris82]

We are setting up a production Splunk environment on Linux VM's (RHEL7). I'm not exactly a "server guy" so learning on the fly. I've been given these specs to build the search head and indexer:

Search head specs:
• Intel 64-bit chip architecture
• 16 CPU cores at 2Ghz or greater speed per core.
• 12GB RAM
• 2 x 300GB, 10,000 RPM SAS hard disks, configured in RAID 1 – 800 IOPS(These a splunk specs not sure what we have available)
• A 1Gb Ethernet NIC, optional 2nd NIC for a management network
• A 64-bit Linux

Indexer specs:
• Intel 64-bit chip architecture.
• 12 CPU cores at 2GHz or greater per core.
• 12GB RAM.
• Disk subsystem capable of 800 average IOPS. For details, see the topic Disk subsystem.
• A 1Gb Ethernet NIC, with optional second NIC for a management network.
• A 64-bit Linux
• 4TB RAID 0 for Hot,Warm, Cold Data – 800 IOPs
• 2TB RAID 0 for Archived Data – 400 IOPS

The questions:

1. How many servers are needed given there are forwarders, indexers, search heads, deployment server, and a license master?
2. Would Splunk be installed on opt/splunk? How much space would need to be allocated to the install?
3. Are there "best practices" for naming the mount points...or would you have any suggestions?
4. On the 4TB drive and (2) 300GB drives, is that space dispersed across several hard disks?

Any suggestions you have would be awesome!

[lguinn2]

First off, you should always have at least 1GB memory per core. So for search heads, get 16GB memory. Also, you don't really need 800 IOPS on the search head for disk I/O. This paragraph may be a small variance from the manual...

The disk spec for the indexer is fine - but I don't know where you came up with the 6TB of disk (hot,warm,cold,archive). So I don't know how many search heads and indexers you need. I don't know if that spec is per indexer or for the whole environment.The number of indexers depends on
1 - how much new data Splunk ingests each day
2 - how many simultaneous users and searches are running
3 - whether or not you are using index replication (indexer clustering)
4 - the total amount of disk that you expect Splunk to manage and search

The number of search heads depends on how much searching is happening in the environment.
You only need one deployment server and one license master for the environment.

There are no rules for naming the mount points - just be sure you are consistent within your environment.

I recommend that you read the entire Capacity Planning manual. But as a starting point, you might read the Summary of performance recommendations.