We've unfortunately had a few instances now where for various reasons the splunkd processes crashes or otherwise stops. Unfortunately the splunkweb process keeps running and presenting a login page HTTP/200 so our load balancer keeps it in the pool.
Ideally there would be a config to have splunkweb shutdown then the splunkd process returns HTTP4XX/5XX or timeouts however I can't see that anywhere.
Worst case I could write a shell script and cron it, but I'm trying to avoid that if I can.
Obviously the easy answer would be to stop splunkd from crashing, but I figure it's better to plan for failure.
... View more