In your Docker image /opt is owned by root, so the splunk user cannot write there. Create /opt/splunk if it does not exist and recursively change ownerships chown -RP splunk:splunk /opt/splunk You can also run your image as root so that it can write to /opt as such: docker run -d -e "SPLUNK_START_ARGS=--accept-license" -e "SPLUNK_USER=root" -p image_name:image_version Or you can rebuild your image so that /opt/splunk is created and owned by the splunk user. To do so, ensure that your DockerFile contains the following lines in the RUN command, and rebuild it (the mkdir and chown's are what you really need to focus on): RUN mkdir -p ${SPLUNK_HOME} \ && wget -qO /tmp/${SPLUNK_FILENAME} http://<remote_host>/${SPLUNK_FILENAME} \ && tar xzf /tmp/${SPLUNK_FILENAME} --strip 1 -C ${SPLUNK_HOME} \ && rm /tmp/${SPLUNK_FILENAME} \ && rm /tmp/${SPLUNK_FILENAME}.md5 \ && mkdir -p /var/opt/splunk \ && cp -R ${SPLUNK_HOME}/etc ${SPLUNK_BACKUP_DEFAULT_ETC} \ && rm -fR ${SPLUNK_HOME}/etc \ && chown -R ${SPLUNK_USER}:${SPLUNK_GROUP} ${SPLUNK_HOME} \ && chown -R ${SPLUNK_USER}:${SPLUNK_GROUP} ${SPLUNK_BACKUP_DEFAULT_ETC} \ I've used both methods with success. Rebuilding with the above lines allowed me to deploy to Kubernetes without issue. In my deployment yaml I have set the following under spec/containers: volumeMounts: - mountPath: /opt/splunk/var/run This should get you on the right track. ... View more

To remove the node from the SHC, perform the steps below on that node (while Splunk is running): Remove the member: splunk remove shcluster-member Disable the member: splunk disable shcluster-config Clean the KVStore: splunk clean kvstore --cluster If you want to re-add this member, I would again verify your DNS entry (check for duplicate records and check /etc/hosts if Linux). Then follow this steps to add the member back into the cluster: Execute these commands in sequence on the problem node: splunk stop splunk clean all splunk start Re-initialize the node: splunk init shcluster-config -auth : -mgmt_uri : -replication_port -replication_factor -conf_deploy_fetch_url : -secret -shcluster_label splunk restart Additional documentation can be found here: https://docs.splunk.com/Documentation/Splunk/7.2.5/DistSearch/Addaclustermember#Add_a_member_that_was_previously_removed_from_the_cluster ... View more