About diligentpenguin

diligentpenguin · ‎06-26-2020

The plot thickens… In looking at my splunkd.log file, it dawned on me that nothing was getting generated because Splunk never even started when I tried to start it. So I looked around and learned that one can look at systemd log files. (Let the record show that I figured this out shortly before your suggestion to do basically the same thing. I’m pretty green when it comes to Linux…) The following command shows Splunk systemd entries. (The -b option means since last boot): journalctl -u Splunkd.service -b It reveals a lot of entries, but here’s the key entries that would appear to elucidate my problems: Jun 26 14:31:51 splunk systemd[1]: Started Systemd service file for Splunk, generated by 'splunk enable boot-start'. Jun 26 14:31:51 splunk splunk[42566]: Checking http port [8000]: open Jun 26 14:31:51 splunk splunk[42566]: Checking mgmt port [8089]: open Jun 26 14:31:52 splunk splunk[42566]: Checking appserver port [127.0.0.1:8065]: open Jun 26 14:31:52 splunk splunk[42566]: Checking kvstore port [8191]: open Jun 26 14:31:52 splunk splunk[42609]: Checking configuration... Done. Jun 26 14:31:52 splunk splunk[42617]: homePath='/mnt/splunk/audit/db' of index=_audit on unusable filesystem. Jun 26 14:31:52 splunk splunk[42617]: Checking critical directories... Done Jun 26 14:31:52 splunk splunk[42617]: Checking indexes... Jun 26 14:31:52 splunk splunk[42566]: Validating databases (splunkd validatedb) failed with code '1'. If you cannot resolve the issue(s) above after consulting documentation, please file a case online at http://www.splunk.com/page/submit_issue Jun 26 14:31:52 splunk systemd[1]: Splunkd.service: Main process exited, code=exited, status=10/n/a Jun 26 14:31:52 splunk systemd[1]: Splunkd.service: Failed with result 'exit-code'. Further digging indicates that CIFS is not supported. And neither really is NFS. Here’s a discussion on the issue. And official documentation at the following URL: https://docs.splunk.com/Documentation/Splunk/8.0.4/Installation/Systemrequirements I also found this gem, which sure enough explains that Splunk is designed to not start when it detects unsupported file systems and that you can bypass this check at your own risk with the hilariously named OPTIMISTIC_ABOUT_FILE_LOCKING=1 variable in the splunk-launch.conf file. So now I need to figure out how to move forward. It would seem my options are: 1) Run with CIFS and hope for the best? The data isn't critical, but my learning is going to be royally interrupted when the whole db is corrupted and I have to trash it and start over. 2) Use NFS (which I can, in theory, enable on my NAS.) 3) Attach an external disk via USB to my little NUC and use that for storage? 4) Use the limited space on my SSD drive for hot and warm buckets, see how fast they fill up, and then setup cold and frozen buckets via NFS. I lean toward option 4. Anyway, thanks for the help. I just wanted to write down what I’d found in case it helps someone else.

diligentpenguin · ‎06-26-2020

Because the log file was rather large, here's the output from the current hour only. Right before grabbing the output, I changed the $SPLUNK_DB variable and tried (unsuccessfully again) to start splunk using the same command I referenced above. I was going to attach the file but the forums only allow jpg, gif, and png. Uggh. So then I tried pasting but was denied because it exceeded 20,000 characters. It seems it's just not my day. So here's the partial log, as a Dropbox link. Thanks!

diligentpenguin · ‎06-26-2020

I have encountered a problem where I cannot get the Splunk service to start after changing The $SPLUNK_DB variable in /opt/splunk/etc/splunk-launch.conf. What I’ve tried and further background information: I have verified that the following steps work successfully if the $SPLUNK_DB variable is NOT set. In other words, it defaults to $SPLUNK_HOME/var/lib/splunk systemctl stop Splunkd.service systemctl start Splunkd.service But once I edit the $SPLUNK_DB variable, I cannot get Splunk to start. Likewise, Splunk will not start after reboot if the $SPLUNK_DB is set. It will start after reboot if this variable is not set. The $SPLUNK_DB variable is set to /mnt/splunk, a CIFS share that I have verified is mounted and can be accessed by the system. (For the curious, this is a testing environment for me to learn Splunk. Splunk is installed on a small NUC with a decent processor and RAM but there’s a single consumer SSD drive with limited space. The CIFS share is on a NAS with multiple terabytes of extra space. I know performance won’t be great, but then, neither will the flow of data.) Next I tried switching to the splunk user (because that seems to be the user that owns the files in the /opt/splunk directory), to see if the issue was a permissions problem. I used sudo su - splunk. I verified that I can indeed create, write, and read, and delete files from /mnt/splunk as the splunk user, root user, and my personal user on Linux. Conclusion: it doesn’t seem to be a permissions problem. Curiously, when I changed the conf file while splunk was running, Splunk created a series of directories and subdirectories inside /mnt/splunk. I can see top level directories of audit, authDb, and hashDb. (There’s no data in them as I don’t have Splunk setup to receive any data yet.) I tried the following search of all the log files hoping I would find clues about why this database path was causing me trouble. /opt/splunk/var/log/splunk# cat *.log | grep 'mnt/splunk' It found nothing. (But if I search instead for the default db path, 'var/lib/splunk', I find dozens or hundreds of entries. (So the search works.) I’m at a loss. Are there other steps I should take beyond changing the path to $SPLUNK_DB? Is there anything I can do to understand why Splunk isn’t starting?

diligentpenguin · ‎06-22-2020

Hi Rich. I very much appreciate the suggestion. I requested the developer license on Thursday and still haven't heard back. I will also add that it looks like developer licenses are good for only half a year, not a full year. I'll keep my fingers crossed.

diligentpenguin · ‎06-18-2020

Hi, I'm wanting to setup Splunk in a home lab for testing and learning purposes. Seeing that I'm so new, and that the enterprise trial license lasts for only 60 days, is it possible to start with the free license while I'm trying to learn the basics and then turn on the trial license later when I'll be able to make better use of it?

Posts	5
Solutions	1
Karma Given	1
Karma Received	0
Member Since	‎06-18-2020

Online Status	Offline
Date Last Visited	‎06-26-2020 05:54 PM

Splunk not starting when $SPLUNK_DB changed

Delay Beginning of Enterprise Trial License?

Re: Splunk not starting when $SPLUNK_DB changed

Re: Splunk not starting when $SPLUNK_DB changed

Splunk not starting when $SPLUNK_DB changed

Re: Delay Beginning of Enterprise Trial License?

Delay Beginning of Enterprise Trial License?