When running splunk on a single windows host (i.e. a windows laptop hosting the indexer and search head, that monitors a local directory in which I ocassionally drop log files into)... what exactly happens when one clicks the "Restart Splunk" button.
For example, does it simply execute a series of windows commands like "net stop splunkd", "net stop splunkweb", "net start splunkd" "net start splunkweb"? Or is there more going on behind the scenes?
I ask becuase I modified the permissions on splunkd and splunkw to allow "All Users" to stop/start them. This works great, however when clicking "Restart Splunk" within the GUI, the services do in fact stop... but never start again.
It sounds like the local instances are trying to perform an action that requires elevated privileges. The first thing that comes to mind is that Splunk has a network input that is configured to listen on UDP 514 (or some other low port) and that's where you're bumping into problems. If splunkd cannot bind to that port - for whatever reason - you're going to have problems, upto and including Splunk not starting correctly.
The easiest way to verify that you're trying to listen on a privileged port is to start Splunk with admin privileges, then run these CLI commands. Keep in mind that if you run these commands when Splunk is stopped you'll get a false negative result that shows no ports listening. Splunk has to be running.
splunk list udp
splunk list tcp
Disable any low port inputs that appear in the output of these commands.
On a separate note, the update checking mechanism can be enabled/disabled per your internal policies. Keep in mind that all it does is reach out to a publicly-available website to check for a new version. This doesn't require any particular privileges above and beyond basic user privileges. You can disable it in web.conf. Search for the term "updateCheckerBaseURL" on that web page.
Port lists are as follows:
UDP Ports: 514
TCP Ports: not currently listening to any tcp port
This is a win 08 server box but I allowed port 514 through and ultimately disabled it entirely. I'm wondering if I should scrap this and throw it on a nix box?
Thanks for the tip on the updateCheckerBaseURL, stumbled across a remote ability to enable remote access as well. Like what I'm seeing so far, hopefully I can get some data displayed. Ha!
The problem is indeed related to the user NOT having admin rights on his windows PC. We know this because if we start a CMD window as Admin then run "splunk.exe -restart" it works fine. However, our users are NOT permitted admin rights... so we're trying to figure a work around. One other interesting not is we noticed "splunk -restarted" mentioned "checking for updates"... perhaps if we could prevent it from looking for updates we would not need admin rights? Any idea how to test this?
Interesting. I am under the impression iro your Q that restart does pretty much as you expected in the 2nd para, stopping the services first...and that the restart element kicks everything off WITH due consideration now to apply any mods just made in the configs files. Any messing with them may alter your stability state, but amending user privileges shouldn't figure in that...
Did you do anything else?! 😉