Getting Data In

Splunk not working across Vagrant Synced folder

dmuth1
New Member

I have an interesting problem--I'm on a Mac, and due to an entirely different issue, I can't reliably run Splunk in OS/X Docker implementation.

No problem--I went and spun up a Vagrant instance running CentOS and decided to run Docker there, and run Splunk in Docker. Seems easy enough, but I ran into any interesting problem: data was being ingested (and showed up in real-time searches), but not syncing to disk. Further investigation revealed that when writing to the internal filesystem in the Vagrant container, the issue did not repeat, but if I tried writing over a directory that is synced to the host filesystem, the problem would show up.

Specifically, there are two things I'm seeing. First, entries like these in splunkd.log:

05-19-2019 21:39:25.397 +0000 ERROR StreamGroup - failed to drain remainder total_sz=3 bytes_freed=560 avg_bytes_per_iv=186 sth=0x7f2dde3fdd50: [1558301964, /opt/splunk/var/lib/splunk/defaultdb/db/hot_v1_0, 0x7f2dd8e6a8a0] reason=st_sync failed rc=-6 warm_rc=[-35,1]

Second, when I look in the directory for any bucket, such as defaultdb/ (main) or _internaldb/ (_internal), I see hundreds and hundreds of files with the string .pre in them:

-rw------- 1 root root 2004 May 19 14:44 1558302293-1558302293-9702670806338853527.pre-tsidx

So the data is making it to disk in some form, it's just not searchable.

To reproduce, here's a Vagrantfile:

Vagrant.configure("2") do |config|

config.vm.box = "minimal/centos7"

config.vm.network "forwarded_port",
guest: 8080, host: 8080

config.vm.provider "virtualbox" do
|vb|
vb.memory = "2048"
vb.cpus = 2 end end

You'll need to install Docker, but yum install -y docker && systemctl start docker should suffice.

Then, you'll need to start my (Dockerized) Splunk App:

SPLUNK_PORT=8080 SPLUNK_START_ARGS=--accept-license bash <(curl -s https://raw.githubusercontent.com/dmuth/splunk-network-health-check/master/go.sh)

As soon as Splunk starts up, running ls -l splunk-data/defaultdb/db/hot_v1_0/ will show those files.

I've never seen anything any error like this before (nor has Google, apparently), so any help or pointers would be appreciated. 🙂

This is with Splunk version Splunk 7.2.5 (build 088f49762779).

Thanks!

0 Karma
Get Updates on the Splunk Community!

How I Instrumented a Rust Application Without Knowing Rust

As a technical writer, I often have to edit or create code snippets for Splunk's distributions of ...

Splunk Community Platform Survey

Hey Splunk Community, Starting today, the community platform may prompt you to participate in a survey. The ...

Observability Highlights | November 2022 Newsletter

 November 2022Observability CloudEnd Of Support Extension for SignalFx Smart AgentSplunk is extending the End ...