A new index never triggers a restart. As a rule of thumb you can say - everything possible to configure in the UI does most of the time not trigger a restart (except a few exceptions). https://conf.splunk.com/files/2017/slides/pushing-configuration-bundles-in-an-indexer-cluster.pdf ... View more

Config Explorer https://splunkbase.splunk.com/app/4353/ ... View more

indexes always reside on every indexer in an indexer cluster. what you really mean is to control the data in indexes on each indexer. It would be better to rephrase that question since it is misleading the potential reader without understanding the topic correctly. And btw... to not replicate your data to protect your data is not a good idea at all. the best option is to build 2 clusters to seperate your data physically or use the index permissions as lguinn2 stated already. ... View more

Not true. Each cluster has exactly one master node. The cluster can have any number of peer nodes and search heads ... View more

When you just pull the plug, be aware that searches wont get the full set of data as long the bucket fixing goes on. That might be a consideration depending on the data you deal with. But I had the same experience with 6.5.5. All the indexers to be decom'ed ran out of disk space in their ../run/searchpeers folder and end up with a log message like: ERROR SearchProcessRunner - launcher_thread=0 runSearch exception: PreforkedSearchProcessException: can't create preforked search process: Cannot send after transport endpoint shutdown which at the end caused more trouble than just pulling the plug. ... View more

I recently ran into the same issue, captain elected but missing in member list and didn't respond to other members anymore. A reboot helped, but not for long, cluster changed to unstabil pretty quick again. Started digging deeper and found the dispatch directory filling (+150k directories) and reaper didn't clean up, so I/O went up crazy. I identified a RT scheduled search causing splunk (6.5.5) keeping all the rt_scheduler__nobody* directories. A rewrite of the search fixed it. Cleaning the dispatch and cluster was running fine again. I afraid i spotted a possible bug in this version. ... View more

did you check for selinux enabled or misconfigured? ... View more

In some cases you want (or have) to run an additional UF on a full installation (enterprise runs as splunk, UF as root to collect all system logs). If the full installation is a DS - it wont work. Splunk checks the hostname, localhost and loopback IP to prevent such situations. The rest of the puzzle is yours. ... View more