Splunk on Solid State Disk?

[southeringtonp]

I'm wondering about placing warm/hot indexes on PCIe-attached solid state disk, with rollover to cold on traditional disk-based storage. It seems like this could have huge benefits for performance.

The biggest concern I see is with the number of write operations and the risk of SSD failure. Maybe use a pair of SSDs in RAID1 to reduce the risk?

Are there any best-practice guidelines for this? Anyone have practical experience?

[araitz]

In Splunk 4.3, there is a really good use case for SSD: Bloom Filters!

http://docs.splunk.com/Documentation/Splunk/latest/Indexer/Bloomfilters

[araitz · Accepted solution]

Keep in mind that SSD will likely not increase indexing performance nor increase dense search performance, as sequential read/write performance on SSD versus traditional HDD is not that different.

However, random seeks will be tremendously faster on SSD vs HDD, so needle in the haystack searches should be quite a bit faster on SSD than HDD.

Another concern is the price per GB of SSD vs. HDD. If you can only afford to put your hot DB and/or hot and a few warm DB on SSD, then you really aren't gaining too much performance, as a lot of the most recent data ends up cached in the operating system, the disk controller, or in other places within Splunk.

[twkan]

Are you going to run this setup in a Enterprise environment, or is it more for testing purposes? For Enterprise environment, most of us will go for SLC drives which have higher write duty cycles.

Since you mentioned PCI-e attached SSD, I assume you are looking into something like the OCZ Revo X2 PCI-e card which I am using for my own personal usage. These are MLC drives, so be aware of the shorter MTBF.

I have my other Sandforce SSD MLC drive died on me before, and this is with ~6 months of usage on moderate write cycles.

You also did not mention which Operating System you will be running on. Depending on the type of controller the SSD drive is using, you may need to have TRIM support on your OS to perform the write leveling and garbage collection.

Regardless of all these, once you have tried SSD you will never use traditional hard drives again. The OCZ Revo X2 can give you over 100,000 IOPS, and the performance is staggering.

I would suggest you to do frequent backups and understand the little intricacies of SSD drives such as TRIM and Secure ATA Erase.

Good luck!

[ftk]

Just based on the huge number of read/writes you're going to experience with having splunk on a SSD I would recommend against it -- MTBF should be reached fairly quickly. I think you may be better served putting more fast spindles in a RAID10 in terms of performance and disk life expectancy.