Monitoring Splunk

A FusionIO ioDrive card with splunk?

gfriedmann
Communicator

Has anyone used an ioDrive for their splunk? I'm really curious if anyone can speak to the crazy high IOPS & throughput significantly helping the splunk search. I've seen a benchmark where the crazy fastness only really was apparent when there were multiple threads using the storage heavily. Maybe the equivalent of 4-6 active searches.

My platform might be considered "low log volume & high search concurrency".

I know the usual advice is to scale out to additional indexers/search servers instead of beefing up individual servers. I'm generally cool with that (i'm planning on adding multiple servers). But this is a specific storage IO question for the crowd.

Do you have any numbers or story to tell?

Tags (2)
1 Solution

araitz
Splunk Employee
Splunk Employee

Yes, we have had customers use FusionIO drives with Splunk.

Since indexing is largely sequential writing, you won't see a tremendous benefit. The same can be said generically for dense reporting searches.

However, for needle-in-the-haystack types of searches (this token appears only a handful of times in millions of results), and to a lesser extent semi-sparse searches (this token appears thousands of times in millions of results), you should experience excellent performance due to the random seek performance of such drives.

In general, because they are so small and so expensive, there is not widespread adoption of FusionIO or similar technology with Splunk, but if money is no object then they help as described above.

Update: Splunk 4.3 has a great use case for SSD drives: Bloom Filters!

http://docs.splunk.com/Documentation/Splunk/4.3/SearchReference/Bloomfilters

View solution in original post

silvermail
Path Finder

Prices has came down somewhat, and I am really tempted to get the OCZ RevoDrive x2 card - which is essentially 4 * 60 GB SF-1200 drives running under RAID 0. Would love to do some benchmarking once I have the card.

0 Karma

araitz
Splunk Employee
Splunk Employee

Yes, we have had customers use FusionIO drives with Splunk.

Since indexing is largely sequential writing, you won't see a tremendous benefit. The same can be said generically for dense reporting searches.

However, for needle-in-the-haystack types of searches (this token appears only a handful of times in millions of results), and to a lesser extent semi-sparse searches (this token appears thousands of times in millions of results), you should experience excellent performance due to the random seek performance of such drives.

In general, because they are so small and so expensive, there is not widespread adoption of FusionIO or similar technology with Splunk, but if money is no object then they help as described above.

Update: Splunk 4.3 has a great use case for SSD drives: Bloom Filters!

http://docs.splunk.com/Documentation/Splunk/4.3/SearchReference/Bloomfilters

Get Updates on the Splunk Community!

Improve Your Security Posture

Watch NowImprove Your Security PostureCustomers are at the center of everything we do at Splunk and security ...

Maximize the Value from Microsoft Defender with Splunk

 Watch NowJoin Splunk and Sens Consulting for this Security Edition Tech TalkWho should attend:  Security ...

This Week's Community Digest - Splunk Community Happenings [6.27.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...