topic Re: Will there be ZFS support on Linux for Splunk Enterprise? in Deployment Architecture

Will there be ZFS support on Linux for Splunk Enterprise?

arnedietrichsta — Thu, 01 Oct 2015 22:41:23 GMT

Hello,

When I tried to install Splunk Enterprise, I noticed that Splunk doesn't like ZFS. Given that MongoDB works just fine on top of ZFS, what do I need to do to get it working?

Here is what I get when I run locktest:

/opt/splunk/bin/locktest
Could not create a lock in the SPLUNK_DB directory.
Filesystem type is not supported: buf.f_type = 0x2fc12fc1
If supporting this filesystem type is important to you, please file an Enhancement Request with Splunk Support with the fs info number listed.

Thanks!

-Arne

PS: No, I do not want to create a fs on top of another fs.

Re: Will there be ZFS support on Linux for Splunk Enterprise?

nnmiller — Sun, 04 Oct 2015 12:49:30 GMT

You should submit an enhancement request to support through your Splunk entitlement account rather than via posting to answers.

Short of Splunk adding support for ZoL, Splunk Enterprise is available for Solaris 10 and 11 on x64 hardware. It should also run fine on on OmniOS or other Illumos-based distributions, though I doubt Splunk will offer support if you have issues on these distributions.

Re: Will there be ZFS support on Linux for Splunk Enterprise?

arnedietrichsta — Tue, 29 Sep 2020 07:31:45 GMT

In the meantime I tried three things:

Disable the lock test (link /opt/splunk/bin/locktest to /bin/true): That didn't work out because (strace to the rescue) there is an additional check in "splunkd validatedb"
Re-import the file system via nfs (changed the zfs mountpoint from /opt/splunk to /opt/splunk-local, exported the directory via nfs, and re-imported the fs to /opt/splunk): That works.
This is what was suggested as a workaround until Splunk officially supports zfs, and it works: echo OPTIMISTIC_ABOUT_FILE_LOCKING = 1 >> /opt/splunk/etc/splunk-launch.conf

Even though the second alternative works, I am not so sure how this will work out in terms of stability and performance. I am now going with the third alternative, and I am quite happy that I don't need to kludge around with nfs.

Re: Will there be ZFS support on Linux for Splunk Enterprise?

nnmiller — Wed, 07 Oct 2015 20:44:05 GMT

It would be an interesting experiment to do a performance test of local native Linux FS vs ZoL mounted via NFS.

Re: Will there be ZFS support on Linux for Splunk Enterprise?

bosburn_splunk — Wed, 07 Oct 2015 20:52:35 GMT

Unfortunately at this time there's no plans in have Splunk running on Linux ZFS. As Nnmiller mentioned, you may want to file an enhancement request.

In addition, your attempts at working around the testing will still cause issues if you have to open a case with Support - please use a supported file system..

Brian

Re: Will there be ZFS support on Linux for Splunk Enterprise?

arnedietrichsta — Fri, 09 Oct 2015 19:27:24 GMT

I heard that ZFS on Linux gets added for the Linux platform in an upcoming release, likely in early 2016.
Solaris, Smartos and Illumos: Yes, those were the days... 😉

Re: Will there be ZFS support on Linux for Splunk Enterprise?

myron_davis — Fri, 12 Feb 2016 20:08:53 GMT

I've been running splunk for years now on ZFS, using many terabytes of disk.

This may be off topic.

I've tested both btrfs and zfs. The version of btrfs I was running (albeit this was a few years ago, maybe fixed?) had massive problems when I tried it with auto snapshot turned on a heavily loaded system. None of those problems existed with zfs.

Even through btrfs is officially supported now, I recommend against it. ZFS with lz4 works very well.

Re: Will there be ZFS support on Linux for Splunk Enterprise?

arnedietrichsta — Fri, 12 Feb 2016 20:26:22 GMT

My problem was that I was using the fuse version of zfs. The kernel module version works like a charm without any workarounds.

Re: Will there be ZFS support on Linux for Splunk Enterprise?

davidpaper — Tue, 26 Apr 2016 17:17:25 GMT

Splunk does support ZFS on Linux as of 6.4.0.

http://docs.splunk.com/Documentation/Splunk/6.4.0/Installation/Systemrequirements#Supported_file_systems

EDIT 10/1/17: This was actually a mistake, and should not have been added to the docs for 6.4.0 or any other Splunk version.

Re: Will there be ZFS support on Linux for Splunk Enterprise?

casial — Fri, 09 Sep 2016 20:20:17 GMT

Spent almost an hour googling and inventing various workarounds... And then I finally found your post. Wouldn't hurt if this simple solution was somewhere more... visible on the splunk website...

Anyway, thaks!

Re: Will there be ZFS support on Linux for Splunk Enterprise?

casial — Fri, 09 Sep 2016 20:29:05 GMT

I downvoted this post because it's wrong, or misleading at least.

Re: Will there be ZFS support on Linux for Splunk Enterprise?

arnedietrichsta — Fri, 09 Sep 2016 20:55:49 GMT

It would be helpful to show which zfs implementation (either kernel zfs or zfs on fuse) is supported.

Re: Will there be ZFS support on Linux for Splunk Enterprise?

arnedietrichsta — Fri, 09 Sep 2016 20:57:48 GMT

Can you please elaborate on that?

Re: Will there be ZFS support on Linux for Splunk Enterprise?

tkomatsubara_sp — Tue, 20 Sep 2016 05:46:24 GMT

Now, ZFS is removed.

Re: Will there be ZFS support on Linux for Splunk Enterprise?

denningsrogue — Sun, 24 Sep 2017 17:12:52 GMT

I downvoted this post because it's incorrect.

Re: Will there be ZFS support on Linux for Splunk Enterprise?

denningsrogue — Sun, 24 Sep 2017 17:14:56 GMT

I downvoted this post because it's incorrect. zfs is not listed as supported under linux.

Re: Will there be ZFS support on Linux for Splunk Enterprise?

davidpaper — Sun, 01 Oct 2017 18:34:19 GMT

This was mistakenly added to the docs, and was removed when it was discovered that QA had not completed tested Splunk on top of ZFS.