My two pennies worth. I have the Debian package installed at home lab and it seems to use systemd as default now. If I restart splunkd as my install user (which is called siem), I am prompted for root password, then a message says I have to restart as root using systemctl. Back when I used init instead it was important to restart splunk as the installation user, siem, otherwise splunk would not start properly, I think because somewhere under the installation tree under /opt/splunk, ownership of a file had changed, (lock file?). I say that because a "chown -R siem:siem /opt/splunk" fixed that issue and siem user could restart splunk again. This is a common issue for us in production and was caused by others upgrading systems and the way they shutdown and start the services, being none the wiser that this would then cause an issue with the Splunk installation. (These are rpm based systems still using init) Similar issue if someone installs splunk as the default user (splunk), siem user could not start splunk until "chown -R siem:siem /opt/splunk" So I wonder if systemd is causing a similar issue, as it appears to be forcing the Splunk service to be started as root and not the user that splunk was installed under. And if remotely restarting, perhaps a prompt for root password is not being seen, so Splunk cannot restart? Maybe an expect script over ssh a remote solution? but not ideal. Maybe sudo is the answer, but that will be a whole lot of servers to manage, does not fit in with the companies security policy, and getting root password is an absolute pain procedure wise. We run a tight ship. I'm hoping I can force a legacy startup until splunk can advise how to install Splunk Enterprise under a specific user and be able to restart Splunk when we need to as that user. Otherwise Splunk just becomes a lump painting us into a corner. Fortunately we are still using init in production, I hope it stays that way. ... View more

I know its old, but you shouldnt be running splunk as root. Install as a splunk user. Then if this issue arises, do. As Root: chown -R splukuser:splunkuser /opt/splunk/ Then as splunkuser $ splunk start ... View more

OK it appears to be working, I am getting disk space used up, but its not the index data. the index data is behaving as it should and now going to the $SPLUNK_DB as specified in splunk-launch.conf, my bad. It looks like I have spool data taking up space, so I probably have a different issue to look at. Thanks. ... View more

In case it helps. We had a similar issue with opsec grabbing data every 30 seconds. There was too much data to process, before the data was processed a new process was started, using up cpu. We increased the time between grabbing the data to 300 seconds. ... View more