I've successfully run a Splunk instance using the splunk-provided run command. I then made a compatible docker compose version of the same command. It runs fine. The issue comes when i want to persist the volume mounts. The splunk image creates two volumes:
/opt/splunk/etc
/opt/splunk/var
So I added volume mounts to my compose file:
volumes:
- /local/path/for/persistence:/opt/splunk/var
- /local/path/for/persistence:/opt/splunk/etc
Now the container fails with output:
fatal: [localhost]: FAILED! => {"changed": false, "cmd": ["/opt/splunk/bin/splunk", "start", "--accept-license", "--answer-yes", "--no-prompt"], "delta": "0:00:03.109600", "end": "2019-05-15 19:46:49.719364", "msg": "non-zero return code", "rc": 10, "start": "2019-05-15 19:46:46.609764", "stderr": "homePath='/opt/splunk/var/lib/splunk/audit/db' of index=_audit on unusable filesystem.\nValidating databases (splunkd validatedb) failed with code '1'. If you cannot resolve the issue(s) above after consulting documentation, please file a case online at http://www.splunk.com/page/submit_issue", "stderr_lines": ["homePath='/opt/splunk/var/lib/splunk/audit/db' of index=_audit on unusable filesystem.", "Validating databases (splunkd validatedb) failed with code '1'. If you cannot resolve the issue(s) above after consulting documentation, please file a case online at http://www.splunk.com/page/submit_issue"], "stdout": "\nSplunk> Finding your faults, just like mom.\n\nChecking prerequisites...\n\tChecking http port [8000]: open\n\tChecking mgmt port [8089]: open\n\tChecking appserver port [127.0.0.1:8065]: open\n\tChecking kvstore port [8191]: open\n\tChecking configuration... Done.\nNew certs have been generated in '/opt/splunk/etc/auth'.\n\tChecking critical directories...\tDone\n\tChecking indexes...\n\t\tCreating: /opt/splunk/var/run/splunk/appserver/i18n\n\t\tCreating: /opt/splunk/var/run/splunk/appserver/modules/static/css\n\t\tCreating: /opt/splunk/var/run/splunk/upload\n\t\tCreating: /opt/splunk/var/spool/splunk\n\t\tCreating: /opt/splunk/var/spool/dirmoncache\n\t\tCreating: /opt/splunk/var/lib/splunk/authDb\n\t\tCreating: /opt/splunk/var/lib/splunk/hashDb", "stdout_lines": ["", "Splunk> Finding your faults, just like mom.", "", "Checking prerequisites...", "\tChecking http port [8000]: open", "\tChecking mgmt port [8089]: open", "\tChecking appserver port [127.0.0.1:8065]: open", "\tChecking kvstore port [8191]: open", "\tChecking configuration... Done.", "New certs have been generated in '/opt/splunk/etc/auth'.", "\tChecking critical directories...\tDone", "\tChecking indexes...", "\t\tCreating: /opt/splunk/var/run/splunk/appserver/i18n", "\t\tCreating: /opt/splunk/var/run/splunk/appserver/modules/static/css", "\t\tCreating: /opt/splunk/var/run/splunk/upload", "\t\tCreating: /opt/splunk/var/spool/splunk", "\t\tCreating: /opt/splunk/var/spool/dirmoncache", "\t\tCreating: /opt/splunk/var/lib/splunk/authDb", "\t\tCreating: /opt/splunk/var/lib/splunk/hashDb"]}
I cannot figure out why this will not work. Everything works until I persist the volumes. If I can't persist the data, then running splunk is useless.
Please try
volumes:
- /local/path/for/persistence/var:/opt/splunk/var/
- /local/path/for/persistence/etc:/opt/splunk/etc/
Also if you need a full ansible/docker/splunk-cluster implementation, please have a try at https://github.com/getkub/ansible_docker_splunk
That's not the issue. Docker does not care if that trailing slash is there.
The actual solution is to set OPTIMISTIC_ABOUT_FILE_LOCKING = 1
in the launchconf. It's probably a bug where splunk doesnt recognize the file system, since it's a user space file system (docker uses union) instead of the expected file system (such as ext4, xfs, etx).
it's not about trailing slash, but rather specific directory for var
and etc
Yes, for the launchconf, the problem happens ONLY in MAC i feel. The fix I've provided is during creation of app,
https://github.com/getkub/ansible_docker_splunk/blob/master/ansible/roles/build_splunk_apps/files/de...
Hi Koshyk,
The link to your repository is broken. Any thoughts?
Thanks,
Gary
After re-reading your original comment, I already have var and etc separated. I just didnt translate that into my post.
And this problem is also in linux. I'm not running on a mac. Debian 9
How did you end up fixing this? I'm having the same issues.