We are planning to put Cold storage on our Indexers in order to accommodate more data.
We are planning to keep the data in hot/warm for 90 days which we are using SSD and on RAID 10.
We have 2 TB of license with Splunk ES. We are mainly keeping the data for audit purpose (as most of the CR searches are looking for past 24hrs of data).
For cold storage whether RAID 5 will be enough or do we need RAID 10?
Lots of questions before an accurate answer can be offered (and even then it'll just be a swag). How many indexers for your 2TB/day ingestion? What kind of disks for your RAID 5? Hardware or software RAID? How many scheduled & accelerated data searches on your data and how far back does the average search look back? How many users? etc...
If you're only using 24 hrs of data for the majority of your ES searches, you certainly don't need SSD/RAID10 for 90 days. You can probably do 2-4 weeks at most. If you already have the disk, go with RAID 5 or 6 for your hot/warm - SSD is way more IOPS than you probably need.
You should be fine with RAID 5 for cold since you're not changing the data. RAID 5 suffers most for writes but the way Splunk writes data to cold is in larger chunks. So there will be a burst of writes when a bucket is rolled to cold, and after that, no writes until the next bucket is rolled. If your bucket sizes are large (ie, autohighvolume or 750mb or larger), you'll not have much issue with disk writes. Unless you use absolutely terrible disks, you should be fine with RAID 5.
Total 20 Indexers with 10TB (SSD for hot/warm) with usable space on RAID 10. Currently ingestion rate is 600GB/day.
RAID 5 - SAS disk with 15K RPM
Adhoc searches - 100,000+ over 24hrs
Saved searches - 1088 over 24hrs
Users - 40+
You are WAY over provisioned for the amount of data you're ingesting. Not a bad thing, since your searches will be super fast, but you probably spent way more than necessary.
For ES, you figure a single indexer can ingest 100GB/day with full ES accelerated data searches turned on. I've seen much more, so that's a conservative value.
You don't say how many spindles you have, but you're probably just fine. With SAS 15k drives, as long as you have more than 5 spindles on a RAID 5 you'll be fine for cold data. I guess you also have write caching so I'd say you're fine.
For the SSD, you don't need RAID 10. RAID 10 is for best performance, but SSD alone is way more than you need for your 100GB/day ingestion and search quantity (I'm guessing you didn't mean 100k adhoc searches, since that would mean people typing in the searches - perhaps you meant acceleration?) You could go to RAID 5 and increase your total disk availability with little to no performance issue, or reserve the extra drives for when you need more indexers.