The plot thickens… In looking at my splunkd.log file, it dawned on me that nothing was getting generated because Splunk never even started when I tried to start it. So I looked around and learned that one can look at systemd log files. (Let the record show that I figured this out shortly before your suggestion to do basically the same thing. I’m pretty green when it comes to Linux…) The following command shows Splunk systemd entries. (The -b option means since last boot): journalctl -u Splunkd.service -b It reveals a lot of entries, but here’s the key entries that would appear to elucidate my problems: Jun 26 14:31:51 splunk systemd: Started Systemd service file for Splunk, generated by 'splunk enable boot-start'. Jun 26 14:31:51 splunk splunk: Checking http port : open Jun 26 14:31:51 splunk splunk: Checking mgmt port : open Jun 26 14:31:52 splunk splunk: Checking appserver port [127.0.0.1:8065]: open Jun 26 14:31:52 splunk splunk: Checking kvstore port : open Jun 26 14:31:52 splunk splunk: Checking configuration... Done. Jun 26 14:31:52 splunk splunk: homePath='/mnt/splunk/audit/db' of index=_audit on unusable filesystem. Jun 26 14:31:52 splunk splunk: Checking critical directories... Done Jun 26 14:31:52 splunk splunk: Checking indexes... Jun 26 14:31:52 splunk splunk: Validating databases (splunkd validatedb) failed with code '1'. If you cannot resolve the issue(s) above after consulting documentation, please file a case online at http://www.splunk.com/page/submit_issue Jun 26 14:31:52 splunk systemd: Splunkd.service: Main process exited, code=exited, status=10/n/a Jun 26 14:31:52 splunk systemd: Splunkd.service: Failed with result 'exit-code'. Further digging indicates that CIFS is not supported. And neither really is NFS. Here’s a discussion on the issue. And official documentation at the following URL: https://docs.splunk.com/Documentation/Splunk/8.0.4/Installation/Systemrequirements I also found this gem, which sure enough explains that Splunk is designed to not start when it detects unsupported file systems and that you can bypass this check at your own risk with the hilariously named OPTIMISTIC_ABOUT_FILE_LOCKING=1 variable in the splunk-launch.conf file. So now I need to figure out how to move forward. It would seem my options are: 1) Run with CIFS and hope for the best? The data isn't critical, but my learning is going to be royally interrupted when the whole db is corrupted and I have to trash it and start over. 2) Use NFS (which I can, in theory, enable on my NAS.) 3) Attach an external disk via USB to my little NUC and use that for storage? 4) Use the limited space on my SSD drive for hot and warm buckets, see how fast they fill up, and then setup cold and frozen buckets via NFS. I lean toward option 4. Anyway, thanks for the help. I just wanted to write down what I’d found in case it helps someone else.
... View more
Hi Rich. I very much appreciate the suggestion. I requested the developer license on Thursday and still haven't heard back. I will also add that it looks like developer licenses are good for only half a year, not a full year. I'll keep my fingers crossed.
... View more