Deployment Architecture

Will there be ZFS support on Linux for Splunk Enterprise?

arnedietrichsta
Explorer

Hello,

When I tried to install Splunk Enterprise, I noticed that Splunk doesn't like ZFS. Given that MongoDB works just fine on top of ZFS, what do I need to do to get it working?

Here is what I get when I run locktest:

/opt/splunk/bin/locktest
Could not create a lock in the SPLUNK_DB directory.
Filesystem type is not supported: buf.f_type = 0x2fc12fc1
If supporting this filesystem type is important to you, please file an Enhancement Request with Splunk Support with the fs info number listed.

Thanks!

-Arne

PS: No, I do not want to create a fs on top of another fs.

Tags (2)
1 Solution

arnedietrichsta
Explorer

My problem was that I was using the fuse version of zfs. The kernel module version works like a charm without any workarounds.

View solution in original post

davidpaper
Contributor

Splunk does support ZFS on Linux as of 6.4.0.

http://docs.splunk.com/Documentation/Splunk/6.4.0/Installation/Systemrequirements#Supported_file_sys...

EDIT 10/1/17: This was actually a mistake, and should not have been added to the docs for 6.4.0 or any other Splunk version.

0 Karma

davidpaper
Contributor

This was mistakenly added to the docs, and was removed when it was discovered that QA had not completed tested Splunk on top of ZFS.

0 Karma

denningsrogue
New Member

I downvoted this post because it's incorrect. zfs is not listed as supported under linux.

0 Karma

arnedietrichsta
Explorer

It would be helpful to show which zfs implementation (either kernel zfs or zfs on fuse) is supported.

0 Karma

tkomatsubara_sp
Splunk Employee
Splunk Employee

Now, ZFS is removed.

0 Karma

arnedietrichsta
Explorer

My problem was that I was using the fuse version of zfs. The kernel module version works like a charm without any workarounds.

View solution in original post

denningsrogue
New Member

I downvoted this post because it's incorrect.

0 Karma

casial
Explorer

I downvoted this post because it's wrong, or misleading at least.

0 Karma

arnedietrichsta
Explorer

Can you please elaborate on that?

0 Karma

arnedietrichsta
Explorer

In the meantime I tried three things:

  1. Disable the lock test (link /opt/splunk/bin/locktest to /bin/true): That didn't work out because (strace to the rescue) there is an additional check in "splunkd validatedb"
  2. Re-import the file system via nfs (changed the zfs mountpoint from /opt/splunk to /opt/splunk-local, exported the directory via nfs, and re-imported the fs to /opt/splunk): That works.
  3. This is what was suggested as a workaround until Splunk officially supports zfs, and it works: echo OPTIMISTIC_ABOUT_FILE_LOCKING = 1 >> /opt/splunk/etc/splunk-launch.conf

Even though the second alternative works, I am not so sure how this will work out in terms of stability and performance. I am now going with the third alternative, and I am quite happy that I don't need to kludge around with nfs.

casial
Explorer

Spent almost an hour googling and inventing various workarounds... And then I finally found your post. Wouldn't hurt if this simple solution was somewhere more... visible on the splunk website...

Anyway, thaks!

0 Karma

nnmiller
Contributor

It would be an interesting experiment to do a performance test of local native Linux FS vs ZoL mounted via NFS.

0 Karma

bosburn_splunk
Splunk Employee
Splunk Employee

Unfortunately at this time there's no plans in have Splunk running on Linux ZFS. As Nnmiller mentioned, you may want to file an enhancement request.

In addition, your attempts at working around the testing will still cause issues if you have to open a case with Support - please use a supported file system..

Brian

myron_davis
Path Finder

I've been running splunk for years now on ZFS, using many terabytes of disk.

This may be off topic.

I've tested both btrfs and zfs. The version of btrfs I was running (albeit this was a few years ago, maybe fixed?) had massive problems when I tried it with auto snapshot turned on a heavily loaded system. None of those problems existed with zfs.

Even through btrfs is officially supported now, I recommend against it. ZFS with lz4 works very well.

nnmiller
Contributor

You should submit an enhancement request to support through your Splunk entitlement account rather than via posting to answers.

Short of Splunk adding support for ZoL, Splunk Enterprise is available for Solaris 10 and 11 on x64 hardware. It should also run fine on on OmniOS or other Illumos-based distributions, though I doubt Splunk will offer support if you have issues on these distributions.

arnedietrichsta
Explorer

I heard that ZFS on Linux gets added for the Linux platform in an upcoming release, likely in early 2016.
Solaris, Smartos and Illumos: Yes, those were the days... 😉

0 Karma