Splunk not working across Vagrant Synced folder

dmuth1 — Sun, 19 May 2019 21:50:55 GMT

I have an interesting problem--I'm on a Mac, and due to an entirely different issue, I can't reliably run Splunk in OS/X Docker implementation.

No problem--I went and spun up a Vagrant instance running CentOS and decided to run Docker there, and run Splunk in Docker. Seems easy enough, but I ran into any interesting problem: data was being ingested (and showed up in real-time searches), but not syncing to disk. Further investigation revealed that when writing to the internal filesystem in the Vagrant container, the issue did not repeat, but if I tried writing over a directory that is synced to the host filesystem, the problem would show up.

Specifically, there are two things I'm seeing. First, entries like these in splunkd.log:

05-19-2019 21:39:25.397 +0000 ERROR StreamGroup - failed to drain remainder total_sz=3 bytes_freed=560 avg_bytes_per_iv=186 sth=0x7f2dde3fdd50: [1558301964, /opt/splunk/var/lib/splunk/defaultdb/db/hot_v1_0, 0x7f2dd8e6a8a0] reason=st_sync failed rc=-6 warm_rc=[-35,1]

Second, when I look in the directory for any bucket, such as defaultdb/ (main) or _internaldb/ (_internal), I see hundreds and hundreds of files with the string .pre in them:

-rw------- 1 root root 2004 May 19 14:44 1558302293-1558302293-9702670806338853527.pre-tsidx

So the data is making it to disk in some form, it's just not searchable.

To reproduce, here's a Vagrantfile:

Vagrant.configure("2") do |config|

config.vm.box = "minimal/centos7"

config.vm.network "forwarded_port",
guest: 8080, host: 8080

config.vm.provider "virtualbox" do
|vb|
vb.memory = "2048"
vb.cpus = 2 end end

You'll need to install Docker, but yum install -y docker && systemctl start docker should suffice.

Then, you'll need to start my (Dockerized) Splunk App:

SPLUNK_PORT=8080 SPLUNK_START_ARGS=--accept-license bash <(curl -s https://raw.githubusercontent.com/dmuth/splunk-network-health-check/master/go.sh)

As soon as Splunk starts up, running ls -l splunk-data/defaultdb/db/hot_v1_0/ will show those files.

I've never seen anything any error like this before (nor has Google, apparently), so any help or pointers would be appreciated. 🙂

This is with Splunk version Splunk 7.2.5 (build 088f49762779).

Thanks!

topic Splunk not working across Vagrant Synced folder in Getting Data In

Splunk not working across Vagrant Synced folder