Getting Data In

Splunk not working across Vagrant Synced folder


I have an interesting problem--I'm on a Mac, and due to an entirely different issue, I can't reliably run Splunk in OS/X Docker implementation.

No problem--I went and spun up a Vagrant instance running CentOS and decided to run Docker there, and run Splunk in Docker. Seems easy enough, but I ran into any interesting problem: data was being ingested (and showed up in real-time searches), but not syncing to disk. Further investigation revealed that when writing to the internal filesystem in the Vagrant container, the issue did not repeat, but if I tried writing over a directory that is synced to the host filesystem, the problem would show up.

Specifically, there are two things I'm seeing. First, entries like these in splunkd.log:

05-19-2019 21:39:25.397 +0000 ERROR StreamGroup - failed to drain remainder total_sz=3 bytes_freed=560 avg_bytes_per_iv=186 sth=0x7f2dde3fdd50: [1558301964, /opt/splunk/var/lib/splunk/defaultdb/db/hot_v1_0, 0x7f2dd8e6a8a0] reason=st_sync failed rc=-6 warm_rc=[-35,1]

Second, when I look in the directory for any bucket, such as defaultdb/ (main) or _internaldb/ (_internal), I see hundreds and hundreds of files with the string .pre in them:

-rw------- 1 root root 2004 May 19 14:44 1558302293-1558302293-9702670806338853527.pre-tsidx

So the data is making it to disk in some form, it's just not searchable.

To reproduce, here's a Vagrantfile:

Vagrant.configure("2") do |config| = "minimal/centos7" "forwarded_port",
guest: 8080, host: 8080

config.vm.provider "virtualbox" do
vb.memory = "2048"
vb.cpus = 2 end end

You'll need to install Docker, but yum install -y docker && systemctl start docker should suffice.

Then, you'll need to start my (Dockerized) Splunk App:

SPLUNK_PORT=8080 SPLUNK_START_ARGS=--accept-license bash <(curl -s

As soon as Splunk starts up, running ls -l splunk-data/defaultdb/db/hot_v1_0/ will show those files.

I've never seen anything any error like this before (nor has Google, apparently), so any help or pointers would be appreciated. 🙂

This is with Splunk version Splunk 7.2.5 (build 088f49762779).


0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!