Splunk Enterprise

Splunk Docker Failing when specifying volume mounts

kparsons
New Member

I've successfully run a Splunk instance using the splunk-provided run command. I then made a compatible docker compose version of the same command. It runs fine. The issue comes when i want to persist the volume mounts. The splunk image creates two volumes:

/opt/splunk/etc
/opt/splunk/var

So I added volume mounts to my compose file:

volumes:
  - /local/path/for/persistence:/opt/splunk/var
  - /local/path/for/persistence:/opt/splunk/etc

Now the container fails with output:

fatal: [localhost]: FAILED! => {"changed": false, "cmd": ["/opt/splunk/bin/splunk", "start", "--accept-license", "--answer-yes", "--no-prompt"], "delta": "0:00:03.109600", "end": "2019-05-15 19:46:49.719364", "msg": "non-zero return code", "rc": 10, "start": "2019-05-15 19:46:46.609764", "stderr": "homePath='/opt/splunk/var/lib/splunk/audit/db' of index=_audit on unusable filesystem.\nValidating databases (splunkd validatedb) failed with code '1'.  If you cannot resolve the issue(s) above after consulting documentation, please file a case online at http://www.splunk.com/page/submit_issue", "stderr_lines": ["homePath='/opt/splunk/var/lib/splunk/audit/db' of index=_audit on unusable filesystem.", "Validating databases (splunkd validatedb) failed with code '1'.  If you cannot resolve the issue(s) above after consulting documentation, please file a case online at http://www.splunk.com/page/submit_issue"], "stdout": "\nSplunk> Finding your faults, just like mom.\n\nChecking prerequisites...\n\tChecking http port [8000]: open\n\tChecking mgmt port [8089]: open\n\tChecking appserver port [127.0.0.1:8065]: open\n\tChecking kvstore port [8191]: open\n\tChecking configuration...  Done.\nNew certs have been generated in '/opt/splunk/etc/auth'.\n\tChecking critical directories...\tDone\n\tChecking indexes...\n\t\tCreating: /opt/splunk/var/run/splunk/appserver/i18n\n\t\tCreating: /opt/splunk/var/run/splunk/appserver/modules/static/css\n\t\tCreating: /opt/splunk/var/run/splunk/upload\n\t\tCreating: /opt/splunk/var/spool/splunk\n\t\tCreating: /opt/splunk/var/spool/dirmoncache\n\t\tCreating: /opt/splunk/var/lib/splunk/authDb\n\t\tCreating: /opt/splunk/var/lib/splunk/hashDb", "stdout_lines": ["", "Splunk> Finding your faults, just like mom.", "", "Checking prerequisites...", "\tChecking http port [8000]: open", "\tChecking mgmt port [8089]: open", "\tChecking appserver port [127.0.0.1:8065]: open", "\tChecking kvstore port [8191]: open", "\tChecking configuration...  Done.", "New certs have been generated in '/opt/splunk/etc/auth'.", "\tChecking critical directories...\tDone", "\tChecking indexes...", "\t\tCreating: /opt/splunk/var/run/splunk/appserver/i18n", "\t\tCreating: /opt/splunk/var/run/splunk/appserver/modules/static/css", "\t\tCreating: /opt/splunk/var/run/splunk/upload", "\t\tCreating: /opt/splunk/var/spool/splunk", "\t\tCreating: /opt/splunk/var/spool/dirmoncache", "\t\tCreating: /opt/splunk/var/lib/splunk/authDb", "\t\tCreating: /opt/splunk/var/lib/splunk/hashDb"]}

I cannot figure out why this will not work. Everything works until I persist the volumes. If I can't persist the data, then running splunk is useless.

Labels (1)
0 Karma

koshyk
Super Champion

Please try

volumes:
   - /local/path/for/persistence/var:/opt/splunk/var/
   - /local/path/for/persistence/etc:/opt/splunk/etc/

Also if you need a full ansible/docker/splunk-cluster implementation, please have a try at https://github.com/getkub/ansible_docker_splunk

0 Karma

kparsons
New Member

That's not the issue. Docker does not care if that trailing slash is there.

The actual solution is to set OPTIMISTIC_ABOUT_FILE_LOCKING = 1 in the launchconf. It's probably a bug where splunk doesnt recognize the file system, since it's a user space file system (docker uses union) instead of the expected file system (such as ext4, xfs, etx).

0 Karma

koshyk
Super Champion

it's not about trailing slash, but rather specific directory for var and etc

Yes, for the launchconf, the problem happens ONLY in MAC i feel. The fix I've provided is during creation of app,
https://github.com/getkub/ansible_docker_splunk/blob/master/ansible/roles/build_splunk_apps/files/de...

0 Karma

gstultz_splunk
Splunk Employee
Splunk Employee

Hi Koshyk, 

The link to your repository is broken.  Any thoughts?

Thanks,

Gary

0 Karma

kparsons
New Member

After re-reading your original comment, I already have var and etc separated. I just didnt translate that into my post.

And this problem is also in linux. I'm not running on a mac. Debian 9

0 Karma

miburo
Explorer

How did you end up fixing this? I'm having the same issues.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...