I have an interesting problem--I'm on a Mac, and due to an entirely different issue, I can't reliably run Splunk in OS/X Docker implementation.
No problem--I went and spun up a Vagrant instance running CentOS and decided to run Docker there, and run Splunk in Docker. Seems easy enough, but I ran into any interesting problem: data was being ingested (and showed up in real-time searches), but not syncing to disk. Further investigation revealed that when writing to the internal filesystem in the Vagrant container, the issue did not repeat, but if I tried writing over a directory that is synced to the host filesystem, the problem would show up.
Specifically, there are two things I'm seeing. First, entries like these in splunkd.log:
05-19-2019 21:39:25.397 +0000 ERROR StreamGroup - failed to drain remainder total_sz=3 bytes_freed=560 avg_bytes_per_iv=186 sth=0x7f2dde3fdd50: [1558301964, /opt/splunk/var/lib/splunk/defaultdb/db/hot_v1_0, 0x7f2dd8e6a8a0] reason=st_sync failed rc=-6 warm_rc=[-35,1]
Second, when I look in the directory for any bucket, such as defaultdb/ (main) or _internaldb/ (_internal), I see hundreds and hundreds of files with the string .pre in them:
-rw------- 1 root root 2004 May 19 14:44 1558302293-1558302293-9702670806338853527.pre-tsidx
So the data is making it to disk in some form, it's just not searchable.
To reproduce, here's a Vagrantfile:
Vagrant.configure("2") do |config|
config.vm.box = "minimal/centos7"
config.vm.network "forwarded_port",
guest: 8080, host: 8080
config.vm.provider "virtualbox" do
|vb|
vb.memory = "2048"
vb.cpus = 2 end end
You'll need to install Docker, but yum install -y docker && systemctl start docker should suffice.
Then, you'll need to start my (Dockerized) Splunk App:
SPLUNK_PORT=8080 SPLUNK_START_ARGS=--accept-license bash <(curl -s https://raw.githubusercontent.com/dmuth/splunk-network-health-check/master/go.sh)
As soon as Splunk starts up, running ls -l splunk-data/defaultdb/db/hot_v1_0/ will show those files.
I've never seen anything any error like this before (nor has Google, apparently), so any help or pointers would be appreciated. 🙂
This is with Splunk version Splunk 7.2.5 (build 088f49762779).
Thanks!
