I just tried this out myself. Just comment out this line in $SPLUNK_HOME/etc/system/default/authorize.conf and restart Splunk. # [capability::delete_by_keyword] The capability will no longer exist for the can_delete role, and you won't be able to assign it to any other role in the Splunk UI. ... View more

The forwarders behavior "out of the box" is to forward data to whatever indexers that are available. In you scenario your secondary indexer could be turned off by default (i.e. not running) - and with some script check if your primary indexer is not running. If the primary indexer is not running - then start the secondary indexer. Seen from the forwarder point of view - this is a "no-brainer" as it will buffer any inputs if your primary indexer dies until it can deliver to your secondary indexer (and vice versa). The problem with this approach might be if you need to search the data from your primary indexer if this becomes unavailable. I have not tried this my self - but this should work. Another approach is to deploy a network load-balancer in front of both indexers, and let this device send data to either your primary or your secondary indexer. ... View more