Update: So all of this was written about Splunk 4.x. So keep that in mind.
For what it's worth. I've found the watchdog "feature" to be more trouble than it's worth, at least on Linux, I haven't tried it on other systems. I've run into situation where it has started up a second instance of splunk because it thought
splunkd was down. (Which was technically true, I suppose) but it was down because
splunk restart was run. So I ended up with two instances of
splunkd concurrently, which ultimately ended up with some index corruption because of it. This happened more than once, or I would have written it off as a fluke.
Here is a script I've been using on a forwarder that, for whatever reason, splunkd seems to crash on quite frequently on that box (every couple months it will crash a couple of times in a row).
**WARNING:** Use this script at your own risks. It's only been tested on one system so far and it could possibly do "bad things". It's also pretty dumb, but it does seem to be slightly smarter than the
splunkmonprocess, at least in my experience. By your mileage may vary. You've been warned!
#!/bin/bash # TODO: Add some kind of runlevel checking or something; make sure we aren't trying to startup whenever the system is poweroff down for example. For now we are just going to risk it. user=splunk proc=splunkd SPLUNK_HOME=/opt/splunk MAINT_FILE=$SPLUNK_HOME/disabled LOGGER="logger -t check_splunkd.sh -s" if [ -f $MAINT_FILE ] then echo "Splunk has been shutdown for maintenance mode (remove $MAINT_FILE) to re-enable automatic splunk restarting." | $LOGGER exit 0; fi # Wait 50 second to see if splunkd is really down (and not just restarting) if ! pgrep -u $user $proc > /dev/null then echo "Splunkd appears to be down." | $LOGGER i=0 while ! pgrep -u $user $proc > /dev/null do let i+=1 if [ $i -gt 10 ]; then break; fi echo "Loop $i" sleep 5 done if [ $i -gt 10 ]; then # Splunkd is not running. Trying to start it up echo "Splunkd still not running. Attempting to start!" | $LOGGER su $user -c "$SPLUNK_HOME/bin/splunk start splunkd" RETVAL=$? echo "Splunk started with RETVAL=$RETVAL" | $LOGGER else echo "Splunkd is now running.... Perhaps splunk was being restarted (i=$i)" | $LOGGER fi fi
I then schedule this script to run every 10 minutes (which is good enough for my needs)
*/10 * * * * root /usr/local/sbin/check_splunkd.sh
If you want to temporarily disable this functionality, like during a splunk upgrade. Simply run:
To re-enable it, simply remove the file: