topic Re: Splunk Docker Failing when specifying volume mounts in Splunk Enterprise

Splunk Docker Failing when specifying volume mounts

kparsons — Wed, 10 Jun 2020 14:52:58 GMT

I've successfully run a Splunk instance using the splunk-provided run command. I then made a compatible docker compose version of the same command. It runs fine. The issue comes when i want to persist the volume mounts. The splunk image creates two volumes:

/opt/splunk/etc
/opt/splunk/var

So I added volume mounts to my compose file:

volumes:
  - /local/path/for/persistence:/opt/splunk/var
  - /local/path/for/persistence:/opt/splunk/etc

Now the container fails with output:

fatal: [localhost]: FAILED! => {"changed": false, "cmd": ["/opt/splunk/bin/splunk", "start", "--accept-license", "--answer-yes", "--no-prompt"], "delta": "0:00:03.109600", "end": "2019-05-15 19:46:49.719364", "msg": "non-zero return code", "rc": 10, "start": "2019-05-15 19:46:46.609764", "stderr": "homePath='/opt/splunk/var/lib/splunk/audit/db' of index=_audit on unusable filesystem.\nValidating databases (splunkd validatedb) failed with code '1'.  If you cannot resolve the issue(s) above after consulting documentation, please file a case online at http://www.splunk.com/page/submit_issue", "stderr_lines": ["homePath='/opt/splunk/var/lib/splunk/audit/db' of index=_audit on unusable filesystem.", "Validating databases (splunkd validatedb) failed with code '1'.  If you cannot resolve the issue(s) above after consulting documentation, please file a case online at http://www.splunk.com/page/submit_issue"], "stdout": "\nSplunk> Finding your faults, just like mom.\n\nChecking prerequisites...\n\tChecking http port [8000]: open\n\tChecking mgmt port [8089]: open\n\tChecking appserver port [127.0.0.1:8065]: open\n\tChecking kvstore port [8191]: open\n\tChecking configuration...  Done.\nNew certs have been generated in '/opt/splunk/etc/auth'.\n\tChecking critical directories...\tDone\n\tChecking indexes...\n\t\tCreating: /opt/splunk/var/run/splunk/appserver/i18n\n\t\tCreating: /opt/splunk/var/run/splunk/appserver/modules/static/css\n\t\tCreating: /opt/splunk/var/run/splunk/upload\n\t\tCreating: /opt/splunk/var/spool/splunk\n\t\tCreating: /opt/splunk/var/spool/dirmoncache\n\t\tCreating: /opt/splunk/var/lib/splunk/authDb\n\t\tCreating: /opt/splunk/var/lib/splunk/hashDb", "stdout_lines": ["", "Splunk> Finding your faults, just like mom.", "", "Checking prerequisites...", "\tChecking http port [8000]: open", "\tChecking mgmt port [8089]: open", "\tChecking appserver port [127.0.0.1:8065]: open", "\tChecking kvstore port [8191]: open", "\tChecking configuration...  Done.", "New certs have been generated in '/opt/splunk/etc/auth'.", "\tChecking critical directories...\tDone", "\tChecking indexes...", "\t\tCreating: /opt/splunk/var/run/splunk/appserver/i18n", "\t\tCreating: /opt/splunk/var/run/splunk/appserver/modules/static/css", "\t\tCreating: /opt/splunk/var/run/splunk/upload", "\t\tCreating: /opt/splunk/var/spool/splunk", "\t\tCreating: /opt/splunk/var/spool/dirmoncache", "\t\tCreating: /opt/splunk/var/lib/splunk/authDb", "\t\tCreating: /opt/splunk/var/lib/splunk/hashDb"]}

I cannot figure out why this will not work. Everything works until I persist the volumes. If I can't persist the data, then running splunk is useless.

Re: Splunk Docker Failing when specifying volume mounts

koshyk — Wed, 15 May 2019 20:31:56 GMT

Please try

volumes:
   - /local/path/for/persistence/var:/opt/splunk/var/
   - /local/path/for/persistence/etc:/opt/splunk/etc/

Also if you need a full ansible/docker/splunk-cluster implementation, please have a try at https://github.com/getkub/ansible_docker_splunk

Re: Splunk Docker Failing when specifying volume mounts

kparsons — Wed, 15 May 2019 20:57:58 GMT

That's not the issue. Docker does not care if that trailing slash is there.

The actual solution is to set OPTIMISTIC_ABOUT_FILE_LOCKING = 1 in the launchconf. It's probably a bug where splunk doesnt recognize the file system, since it's a user space file system (docker uses union) instead of the expected file system (such as ext4, xfs, etx).

Re: Splunk Docker Failing when specifying volume mounts

koshyk — Wed, 15 May 2019 21:15:30 GMT

it's not about trailing slash, but rather specific directory for var and etc

Yes, for the launchconf, the problem happens ONLY in MAC i feel. The fix I've provided is during creation of app,
https://github.com/getkub/ansible_docker_splunk/blob/master/ansible/roles/build_splunk_apps/files/dev/MY_all_env_specific/bin/splunk_set_mac.py

Re: Splunk Docker Failing when specifying volume mounts

kparsons — Wed, 15 May 2019 21:18:43 GMT

After re-reading your original comment, I already have var and etc separated. I just didnt translate that into my post.

And this problem is also in linux. I'm not running on a mac. Debian 9

Re: Splunk Docker Failing when specifying volume mounts

miburo — Thu, 31 Oct 2019 19:26:46 GMT

How did you end up fixing this? I'm having the same issues.

Re: Splunk Docker Failing when specifying volume mounts

gstultz_splunk — Wed, 19 Aug 2020 14:45:51 GMT

Hi Koshyk,

The link to your repository is broken. Any thoughts?

Thanks,

Gary