Monitoring Splunk

A FusionIO ioDrive card with splunk?

gfriedmann
Communicator

Has anyone used an ioDrive for their splunk? I'm really curious if anyone can speak to the crazy high IOPS & throughput significantly helping the splunk search. I've seen a benchmark where the crazy fastness only really was apparent when there were multiple threads using the storage heavily. Maybe the equivalent of 4-6 active searches.

My platform might be considered "low log volume & high search concurrency".

I know the usual advice is to scale out to additional indexers/search servers instead of beefing up individual servers. I'm generally cool with that (i'm planning on adding multiple servers). But this is a specific storage IO question for the crowd.

Do you have any numbers or story to tell?

Tags (2)
1 Solution

araitz
Splunk Employee
Splunk Employee

Yes, we have had customers use FusionIO drives with Splunk.

Since indexing is largely sequential writing, you won't see a tremendous benefit. The same can be said generically for dense reporting searches.

However, for needle-in-the-haystack types of searches (this token appears only a handful of times in millions of results), and to a lesser extent semi-sparse searches (this token appears thousands of times in millions of results), you should experience excellent performance due to the random seek performance of such drives.

In general, because they are so small and so expensive, there is not widespread adoption of FusionIO or similar technology with Splunk, but if money is no object then they help as described above.

Update: Splunk 4.3 has a great use case for SSD drives: Bloom Filters!

http://docs.splunk.com/Documentation/Splunk/4.3/SearchReference/Bloomfilters

View solution in original post

silvermail
Path Finder

Prices has came down somewhat, and I am really tempted to get the OCZ RevoDrive x2 card - which is essentially 4 * 60 GB SF-1200 drives running under RAID 0. Would love to do some benchmarking once I have the card.

0 Karma

araitz
Splunk Employee
Splunk Employee

Yes, we have had customers use FusionIO drives with Splunk.

Since indexing is largely sequential writing, you won't see a tremendous benefit. The same can be said generically for dense reporting searches.

However, for needle-in-the-haystack types of searches (this token appears only a handful of times in millions of results), and to a lesser extent semi-sparse searches (this token appears thousands of times in millions of results), you should experience excellent performance due to the random seek performance of such drives.

In general, because they are so small and so expensive, there is not widespread adoption of FusionIO or similar technology with Splunk, but if money is no object then they help as described above.

Update: Splunk 4.3 has a great use case for SSD drives: Bloom Filters!

http://docs.splunk.com/Documentation/Splunk/4.3/SearchReference/Bloomfilters

Get Updates on the Splunk Community!

Index This | What did the zero say to the eight?

June 2025 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with this month’s ...

Splunk Observability Cloud's AI Assistant in Action Series: Onboarding New Hires & ...

This is the fifth post in the Splunk Observability Cloud’s AI Assistant in Action series that digs into how to ...

Now Playing: Splunk Education Summer Learning Premieres

It’s premiere season, and Splunk Education is rolling out new releases you won’t want to miss. Whether you’re ...