Monitoring Splunk

Recommended filesystem for Centos/Redhat

zscgeek
Path Finder

What filesystem is recommended for maximum performance on centos/redhat 5.x? (64 bit)

We were thinking either EXT3 or XFS as they are what we have used the most but wanted to get the official splunk recommendation. This will be for our indexing servers that will be doing high volume indexing and searching and storing data for long periods of time.

Overall hardware config will be:

  • 2x Quad Core Intel CPU (L5410)
  • 16GB Ram
  • 16x1TB SATA drives (7200 rpm) in an eSAS chassis.
  • LSI RAID 5 or 10 depending on performance
Tags (2)
0 Karma
1 Solution

Jeremiah
Motivator

Raid 10 is supposed to have the best performance, and I've heard that recommended from several sources. I'm not sure the filesystem makes as significant a difference as long as its supported (someone please correct me if I'm wrong.).

http://www.splunk.com/wiki/Community:HardwareTuningFactors

View solution in original post

Jeremiah
Motivator

Raid 10 is supposed to have the best performance, and I've heard that recommended from several sources. I'm not sure the filesystem makes as significant a difference as long as its supported (someone please correct me if I'm wrong.).

http://www.splunk.com/wiki/Community:HardwareTuningFactors

View solution in original post

Lowell
Super Champion

I use XFS for my primary splunk partition ($SPLUNK_HOME/), and then I use ext3 for the $SPLUNK_HOME/var/run partition (so that all the search jobs and temporary files that splunk creates doesn't cause fragmentation issues with the actual indexing process). (There is one minor bug with outputlookup because of my separate partitions, which has been reported and should be fixed soon.) I've also got all my partitions on top of LVM for easy partition reallocation.

0 Karma

mcluver
Path Finder

Is it absolutely necessary to do it this way if we're thinking about going with XFS on an Indexer?

0 Karma

zscgeek
Path Finder

My concern with filesystems is that some handle large number of files better then others. Splunk can end up creating insane numbers of files when you get to holding several TB worth of logs in the raw dirs. EXT3 is the OS default of course but I would not mind knowing if people have hit limitations with it.

0 Karma
Register for .conf21 Now! Go Vegas or Go Virtual!

How will you .conf21? You decide! Go in-person in Las Vegas, 10/18-10/21, or go online with .conf21 Virtual, 10/19-10/20.