Getting Data In

Splunk not working across Vagrant Synced folder

dmuth1
New Member

I have an interesting problem--I'm on a Mac, and due to an entirely different issue, I can't reliably run Splunk in OS/X Docker implementation.

No problem--I went and spun up a Vagrant instance running CentOS and decided to run Docker there, and run Splunk in Docker. Seems easy enough, but I ran into any interesting problem: data was being ingested (and showed up in real-time searches), but not syncing to disk. Further investigation revealed that when writing to the internal filesystem in the Vagrant container, the issue did not repeat, but if I tried writing over a directory that is synced to the host filesystem, the problem would show up.

Specifically, there are two things I'm seeing. First, entries like these in splunkd.log:

05-19-2019 21:39:25.397 +0000 ERROR StreamGroup - failed to drain remainder total_sz=3 bytes_freed=560 avg_bytes_per_iv=186 sth=0x7f2dde3fdd50: [1558301964, /opt/splunk/var/lib/splunk/defaultdb/db/hot_v1_0, 0x7f2dd8e6a8a0] reason=st_sync failed rc=-6 warm_rc=[-35,1]

Second, when I look in the directory for any bucket, such as defaultdb/ (main) or _internaldb/ (_internal), I see hundreds and hundreds of files with the string .pre in them:

-rw------- 1 root root 2004 May 19 14:44 1558302293-1558302293-9702670806338853527.pre-tsidx

So the data is making it to disk in some form, it's just not searchable.

To reproduce, here's a Vagrantfile:

Vagrant.configure("2") do |config|

config.vm.box = "minimal/centos7"

config.vm.network "forwarded_port",
guest: 8080, host: 8080

config.vm.provider "virtualbox" do
|vb|
vb.memory = "2048"
vb.cpus = 2 end end

You'll need to install Docker, but yum install -y docker && systemctl start docker should suffice.

Then, you'll need to start my (Dockerized) Splunk App:

SPLUNK_PORT=8080 SPLUNK_START_ARGS=--accept-license bash <(curl -s https://raw.githubusercontent.com/dmuth/splunk-network-health-check/master/go.sh)

As soon as Splunk starts up, running ls -l splunk-data/defaultdb/db/hot_v1_0/ will show those files.

I've never seen anything any error like this before (nor has Google, apparently), so any help or pointers would be appreciated. 🙂

This is with Splunk version Splunk 7.2.5 (build 088f49762779).

Thanks!

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...