switching ext3 for xfs

[rotten]

All other things being equal, would we see any performance gains with Splunk if we switch our file system from ext3 to xfs?

Another thread on Splunk answers recommended moving $SPLUNK_HOME/var/run to its own ext3 file system because lots of little files end up there.

However I never saw any real confirmation in that thread as to whether the possible performance gains were worth the effort of switching the file system out on an already running Splunk indexer.

We are always looking for ways to get Splunk to perform a little better....

[Lowell]

I setup our primary splunk partition as XFS simply because it allows for file defragging where as ext3 does not. Any kind of file fragmentation will impact performance because of extra seeks.

With ext4 offering defrag support as well, perhaps that would be an option worth considering; although it's still a bit new, and apparently the defrag support isn't considered stable yet.

Of course, splunk's file usage patterns seem to keep fragmentation fairly low. All the rawdata files can get fragmented, but then get gziped (and thus re-written to disk) which helps keep fragmentation low. Also the *.tsidx files can get fragmented as new content is added to a bucket, but then these files get merged over time, which again triggers the creation of a new file which is much less likely to be fragmented.

You can look at some fragmentation of various files on your ext3 files system using the filefrag tool. You have to run this as root, and it only reports one file at a time, so I often run a command like:

find /opt/splunk/var/lib/splunk -type f | xargs filefrag

[gkanapathy]

I suspect that performance gains from changing file systems will likely be invisible. Most of the disk performance time is vastly and overwhelmingly consumed by having the disk head move to get data from places in different files, and the amount of movement required is not likely to be changed by a change of file systems, i.e., it won't really IOPS, and the number of IOPS for any Splunk search will be basically the same for any file system.

(Of course, things like filesystem-level RAID and LVM and so forth changes this calculation, but isn't relevant for the case of ext vs xfs. The issue then isn't the filesystem per se, but rather the disk layout.)