Archive
Highlighted

Can I run splunk on btrfs?

Explorer

Hello,

I just downloaded splunk today to try it out on a few of our servers, but found out very quickly that it doesn't support btrfs:

Filesystem type is not supported: buf.f_type = 0x9123683e
  1. Why does splunk care about the file system anyway?
  2. Is there a way to "force" btrfs support, maybe with reduced functionality?
  3. Is official support for btrfs planned?

The output of locktest looks like this:

~/splunk]% bin/locktest                                 
Could not create a lock in the SPLUNK_DB directory.
Filesystem type is not supported: buf.f_type = 0x9123683e
If supporting this filesystem type is important to you, please file an Enhancement Request with Splunk Support with the fs info number listed.

~/splunk]% ls $SPLUNK_DB 
audit/  authDb/  blockSignature/  defaultdb/  fishbucket/  hashDb/  historydb/  _internaldb/  sample/  summarydb/  test.ijKHJ9  test.R0jT0h  test.T65SU0

Output of strace:

~/splunk]% strace bin/locktest
execve("bin/locktest", ["bin/locktest"], [/* 32 vars */]) = 0
brk(0)                                  = 0x245b000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f8ad5120000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY)      = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=27312, ...}) = 0
mmap(NULL, 27312, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f8ad5119000
close(3)                                = 0
open("/lib64/libc.so.6", O_RDONLY)      = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0p\355\1\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=1832712, ...}) = 0
mmap(NULL, 3664040, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f8ad4b84000
mprotect(0x7f8ad4cf9000, 2097152, PROT_NONE) = 0
mmap(0x7f8ad4ef9000, 20480, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x175000) = 0x7f8ad4ef9000
mmap(0x7f8ad4efe000, 18600, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f8ad4efe000
close(3)                                = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f8ad5118000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f8ad5117000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f8ad5116000
arch_prctl(ARCH_SET_FS, 0x7f8ad5117700) = 0
mprotect(0x7f8ad4ef9000, 16384, PROT_READ) = 0
mprotect(0x7f8ad5121000, 4096, PROT_READ) = 0
munmap(0x7f8ad5119000, 27312)           = 0
stat("/home/xx/splunk/var/lib/splunk", {st_mode=S_IFDIR|0711, st_size=258, ...}) = 0
umask(0777)                             = 022
umask(066)                              = 0777
gettimeofday({1290061806, 661323}, NULL) = 0
getpid()                                = 15124
open("/home/xx/splunk/var/lib/splunk/test.dHc3Nt", O_RDWR|O_CREAT|O_EXCL, 0600) = 3
statfs("/home/xx/splunk/var/lib/splunk/test.dHc3Nt", {f_type=0x9123683e,     f_bsize=4096, f_blocks=2228224, f_bfree=1430113, f_bavail=1128033, f_files=0, f_ffree=0, f_fsid={196237592, 245698777}, f_namelen=255, f_frsize=4096}) = 0
statfs("/home/xx/splunk/var/lib/splunk/test.dHc3Nt", {f_type=0x9123683e, f_bsize=4096, f_blocks=2228224, f_bfree=1430113, f_bavail=1128033, f_files=0, f_ffree=0, f_fsid={196237592, 245698777}, f_namelen=255, f_frsize=4096}) = 0
statfs("/home/xx/splunk/var/lib/splunk", {f_type=0x9123683e, f_bsize=4096, f_blocks=2228224, f_bfree=1430113, f_bavail=1128033, f_files=0, f_ffree=0, f_fsid={196237592, 245698777}, f_namelen=255, f_frsize=4096}) = 0
write(2, "Could not create a lock in the S"..., 52Could not create a lock in the SPLUNK_DB directory.) = 52
statfs("/home/xx/splunk/var/lib/splunk/test.dHc3Nt", {f_type=0x9123683e, f_bsize=4096, f_blocks=2228224, f_bfree=1430113, f_bavail=1128033, f_files=0, f_ffree=0, f_fsid={196237592, 245698777}, f_namelen=255, f_frsize=4096}) = 0
statfs("/home/xx/splunk/var/lib/splunk/test.dHc3Nt", {f_type=0x9123683e, f_bsize=4096, f_blocks=2228224, f_bfree=1430113, f_bavail=1128033, f_files=0, f_ffree=0, f_fsid={196237592, 245698777}, f_namelen=255, f_frsize=4096}) = 0
write(2, "Filesystem type is not supported"..., 201Filesystem type is not supported: buf.f_type = 0x9123683e
If supporting this filesystem type is important to you, please file an Enhancement Request with Splunk Support with the fs info number listed.) = 201
exit_group(9)    
Tags (1)
Highlighted

Re: Can I run splunk on btrfs?

SplunkTrust
SplunkTrust

I don't know precisely why Splunk threw an unsupported message about your btrfs - but it is usually very picky about the underlying filesystem's locking semantics. What does the Splunk 'locktest' command say about your btrfs?

Highlighted

Re: Can I run splunk on btrfs?

Explorer

Thanks for the response! I just updated the question with the output of locktest.

0 Karma
Highlighted

Re: Can I run splunk on btrfs?

SplunkTrust
SplunkTrust

You might try running locktest under strace, and see if Splunk is hitting a specific error with its lock testing for btrfs, or if it is excluding this filesystem based on its magic value alone. Then you'll know whether to get after the btrfs developers for improved locking or to file an enhancement with Splunk support.

0 Karma
Highlighted

Re: Can I run splunk on btrfs?

Explorer

I added the output of strace as well. I'm not sure how to read that, so could you please take a loot?

0 Karma
Highlighted

Re: Can I run splunk on btrfs?

SplunkTrust
SplunkTrust

I don't see any specific errors in your strace related to locktest - I would suggest putting in an ER (enhancement request) at this point. If you are working with someone within sales, make sure they are aware of your requirement. See http://answers.splunk.com/questions/4844/how-can-i-submit-an-enhancement-request

Highlighted

Re: Can I run splunk on btrfs?

Explorer

Understood. Thanks for the reply!

0 Karma
Highlighted

Re: Can I run splunk on btrfs?

Engager

Since btrfs is planned to become the default in Fedora 17 and others will surely follow, I think splunk should start to support it as soon as possible.
As a workaround I've loopback-mounted a file formatted with ext4 to /opt/splunk/var and copied the contents of the original directory there. Seems to work just fine so far.

0 Karma
Highlighted

Re: Can I run splunk on btrfs?

Engager

"As a workaround I've loopback-mounted a file formatted with ext4 to /opt/splunk/var and copied the contents of the original directory there. Seems to work just fine so far."

Please provide a "how to" for this workaround as I am new to linux and dont completely follow how this is achieved. Any help appreciated. I created a loopback file system but I am confused about recreating the var directory as an ext4 filesystem since this did not work when following this procedure..http://www.walkernews.net/2007/07/01/create-linux-loopback-file-system-on-disk-file/#comment-15959

Regards, Henry

Highlighted

Re: Can I run splunk on btrfs?

Engager

Hi Henry/Archie(?),

first create a file as big as you think you will need for splunk data

dd if=/dev/zero of=/mydata/splunkfs.img bs=1M count=5000

Then format it with ext4 (sans journal, which is kind of moot in this case):

mkfs.ext4 -O ^has_journal /mydata/splunkfs.img

Add a line to /etc/fstab like this:

/mydata/splunkfs.img  /opt/splunk/var/        auto    loop    0 0

Stop splunk:

/opt/splunk/bin/splunk stop

Then just something like

cd /opt/splunk

mkdir vartmp

mv var/* vartmp

mount var

mv vartmp/* var

/opt/splunk/bin/splunk start