The distinction between HOT/WARM and COLD exists for only one purpose: To allow you to pick the fastest possible disk (and hence the most expensive) for the data you likely search most, which is recent, and pick cheaper mass storage for long term retention (often with relaxed performance requirements). Based on our experience, most environments access data within a 24hr period about >90% of the time, so ensuring that these searches complete as fast as possible is key.
HOT/WARM path is heavily read write, COLD is write once, read many. The only difference between HOT and WARM is that HOT buckets are actively written to, whereas WARM buckets are read-only once they have been created. So: Inbound data->HOT buckets, HOT buckets roll to WARM after a configurable time has elapsed or a configurable number of buckets have been created. WARM buckets roll to COLD after a configurable number or size has been reached. COLD buckets roll to FROZEN after a configurable time has elapsed or size has been reached. See here for more details on that.
If you only have one kind of disk, it doesn't make a difference. I would have one volume for the OS and Application, and another volume for your index data storage. You can create Splunk Volumes to manage space for multiple indices that have the same retention settings as a whole, if you want to. There is no real reason why you can't use multiple disk volumes, but I don't really see the benefit of it either. What if you run out of space on volume1 while volume2 happens to have plenty of space available?
While I understand that you may not have that option, your biggest benefit would be to have a couple of SSDs in each server that you can use for HOT/WARM, and save your spinning disks for COLD storage. If you just had 1TB of SSD, you'd be able to almost keep a full day's worth of logs in HOT/WARM, removing all I/O contention for >90% of your workload.
HTH
... View more
