As we begin to plan out our deployment of Splunk, one thing is starting to puzzle me, and this is mostly a "should we consider SSD's" type of question. Based on some test data available elsewhere, even consumer level SSD's are capable of writing up to 2+ Petabytes of information before they die.
So, if our plan is to ingest roughly 10 GB of logs into our Splunk service daily. How much extra data should I consider added by indexing, etc done by the Splunk server? An extra 5 GB? 10? 20?
I realize that this is probably a good "it depends" type of answer, but are there any rough ballpark figures to go on? I was not able to find sizing information from the splunk site (other than minimum IOPS to effectively run Splunk). If indexing is minimal (or even equivalent) to the ingest rate, that's still 15GB, 20 or 30 GB per day, something that a "decent" even consumer level SATA SSD can write for decades before wearing out. Even 1TB of extra writes per day of log ingest is "fine", I just have to plan for replacing SSD's more often. Given that SSD's are incredibly cheap for the performance, and given that I already know that SSD's do eventually wear out (though so do HDD's), I'd like to have a plan for "regular" scheduled replacement plans for our storage infrastructure. Like "plan on replacing at a rate of x per year". We do plan on investing in more business/enterprise grade storage, though. I was just using a consumer SSD as a baseline.
Does anyone use SSD's for their entire storage array for Splunk? I imagine this would be a small-ish size (10 GB per day isn't that many logs), so getting something approaching the SSD performance, but in an HDD format means spending a lot of money on massive spindle arrays that I just don't think we can justify.
... View more