Splunk Enterprise
Highlighted

Splunk Docker Failing when specifying volume mounts

New Member

I've successfully run a Splunk instance using the splunk-provided run command. I then made a compatible docker compose version of the same command. It runs fine. The issue comes when i want to persist the volume mounts. The splunk image creates two volumes:

/opt/splunk/etc
/opt/splunk/var

So I added volume mounts to my compose file:

volumes:
  - /local/path/for/persistence:/opt/splunk/var
  - /local/path/for/persistence:/opt/splunk/etc

Now the container fails with output:

fatal: [localhost]: FAILED! => {"changed": false, "cmd": ["/opt/splunk/bin/splunk", "start", "--accept-license", "--answer-yes", "--no-prompt"], "delta": "0:00:03.109600", "end": "2019-05-15 19:46:49.719364", "msg": "non-zero return code", "rc": 10, "start": "2019-05-15 19:46:46.609764", "stderr": "homePath='/opt/splunk/var/lib/splunk/audit/db' of index=_audit on unusable filesystem.\nValidating databases (splunkd validatedb) failed with code '1'.  If you cannot resolve the issue(s) above after consulting documentation, please file a case online at http://www.splunk.com/page/submit_issue", "stderr_lines": ["homePath='/opt/splunk/var/lib/splunk/audit/db' of index=_audit on unusable filesystem.", "Validating databases (splunkd validatedb) failed with code '1'.  If you cannot resolve the issue(s) above after consulting documentation, please file a case online at http://www.splunk.com/page/submit_issue"], "stdout": "\nSplunk> Finding your faults, just like mom.\n\nChecking prerequisites...\n\tChecking http port [8000]: open\n\tChecking mgmt port [8089]: open\n\tChecking appserver port [127.0.0.1:8065]: open\n\tChecking kvstore port [8191]: open\n\tChecking configuration...  Done.\nNew certs have been generated in '/opt/splunk/etc/auth'.\n\tChecking critical directories...\tDone\n\tChecking indexes...\n\t\tCreating: /opt/splunk/var/run/splunk/appserver/i18n\n\t\tCreating: /opt/splunk/var/run/splunk/appserver/modules/static/css\n\t\tCreating: /opt/splunk/var/run/splunk/upload\n\t\tCreating: /opt/splunk/var/spool/splunk\n\t\tCreating: /opt/splunk/var/spool/dirmoncache\n\t\tCreating: /opt/splunk/var/lib/splunk/authDb\n\t\tCreating: /opt/splunk/var/lib/splunk/hashDb", "stdout_lines": ["", "Splunk> Finding your faults, just like mom.", "", "Checking prerequisites...", "\tChecking http port [8000]: open", "\tChecking mgmt port [8089]: open", "\tChecking appserver port [127.0.0.1:8065]: open", "\tChecking kvstore port [8191]: open", "\tChecking configuration...  Done.", "New certs have been generated in '/opt/splunk/etc/auth'.", "\tChecking critical directories...\tDone", "\tChecking indexes...", "\t\tCreating: /opt/splunk/var/run/splunk/appserver/i18n", "\t\tCreating: /opt/splunk/var/run/splunk/appserver/modules/static/css", "\t\tCreating: /opt/splunk/var/run/splunk/upload", "\t\tCreating: /opt/splunk/var/spool/splunk", "\t\tCreating: /opt/splunk/var/spool/dirmoncache", "\t\tCreating: /opt/splunk/var/lib/splunk/authDb", "\t\tCreating: /opt/splunk/var/lib/splunk/hashDb"]}

I cannot figure out why this will not work. Everything works until I persist the volumes. If I can't persist the data, then running splunk is useless.

Labels (1)
0 Karma
Highlighted

Re: Splunk Docker Failing when specifying volume mounts

Super Champion

Please try

volumes:
   - /local/path/for/persistence/var:/opt/splunk/var/
   - /local/path/for/persistence/etc:/opt/splunk/etc/

Also if you need a full ansible/docker/splunk-cluster implementation, please have a try at https://github.com/getkub/ansible_docker_splunk

0 Karma
Highlighted

Re: Splunk Docker Failing when specifying volume mounts

New Member

That's not the issue. Docker does not care if that trailing slash is there.

The actual solution is to set OPTIMISTIC_ABOUT_FILE_LOCKING = 1 in the launchconf. It's probably a bug where splunk doesnt recognize the file system, since it's a user space file system (docker uses union) instead of the expected file system (such as ext4, xfs, etx).

0 Karma
Highlighted

Re: Splunk Docker Failing when specifying volume mounts

Super Champion

it's not about trailing slash, but rather specific directory for var and etc

Yes, for the launchconf, the problem happens ONLY in MAC i feel. The fix I've provided is during creation of app,
https://github.com/getkub/ansible_docker_splunk/blob/master/ansible/roles/build_splunk_apps/files/de...

0 Karma
Highlighted

Re: Splunk Docker Failing when specifying volume mounts

New Member

After re-reading your original comment, I already have var and etc separated. I just didnt translate that into my post.

And this problem is also in linux. I'm not running on a mac. Debian 9

0 Karma
Highlighted

Re: Splunk Docker Failing when specifying volume mounts

Explorer

How did you end up fixing this? I'm having the same issues.

0 Karma