How to start a Spunk Docker container via Dockerfile and adding it as a search peer in one bash script?

[krishanp]

Hello,

I am attempting to start a Splunk docker container (search head) and add it as a search peer to an existing environment all in one bash script but running in an issue. I am able to run each of the two steps separately without a problem but am running into an issue when I attempt to combine them into one script. 

I am able to build my Dockerfile and start the container successfully. I am running the below command to start a container with the name splunk_sh.

docker run -d --rm -it -p 8000:8000 --name splunk_sh dockersplunk:latest

After the container is up, I am also able to successfully add it as a search peer using the following command and script. (A copy of the search_peer.sh script is being copied to my container via Dockerfile.)

# search peer command
docker exec -it splunk_sh sh /opt/splunk/bin/search_peer.sh

search_peer.sh 

#!/bin/bash
sudo /opt/splunk/bin/splunk add search-server https://<indexer_ip>:8089 -auth <user>:<password> -remoteUsername <user> -remotePassword <password>

Running the two above steps separately allows me to start my Splunk container and have it become a search peer. I begin to run into an issue when I try to run a script (docker_search_peer.sh) that includes both steps, starting the splunk_sh container and the search peer command. 

docker_search_peer.sh

#!/bin/bash
docker run -d --rm -it -p 8000:8000 --name splunk_sh dockersplunk:latest
docker exec -it splunk_sh sh /opt/splunk/bin/search_peer.sh

When I run my docker_search_peer.sh script, the container is able to start but is not able to become a search peer. I get the below error:

ERROR: Couldn't determine $SPLUNK_HOME or $SPLUNK_ETC; perhaps one should be set in environment

I've disabled selinux (this was mentioned in a few different posts) but am still running into this issue. I'm not sure how I'm able to run commands/execute scripts separately but not together in one script. Any help or guidance would be much appreciated. 

[Tom_Lundie]

Hi @krishanp,

I can't find that dockersplunk container image (I'm presuming it's a private one) but I'm thinking that one of two things might be going on here.

Either:

A) (most-likely) docker exec is running before Splunk is up and ready. The command started using docker exec only runs while the container’s primary process (PID 1) is running.

B) (less-likely based off your post) search_peer.sh is not running as the user that Splunk is running under.

Either way, make sure that neither of those are happening. Depending on how the dockerfile is structured, it's possible that Splunk is still being configured / starting whilst the search_peer.sh is starting.

If you can, then I would either update the dockerfile to run the search_peer.sh after Splunk is up. If that is not an option here, then you could modify either search_peer.sh or docker_search_peer.sh to wait until Splunk is up before adding the search peer. There are loads of examples out there around this (e.g. you could wait until SplunkWeb is listening on 8000).

Let me know how you get on. Good luck! 

[krishanp]

Thank you @Tom_Lundie! It seems that I was running into scenario A (docker exec is running before Splunk is up and ready). I added a while loop to check my container logs and confirm that the playbook has completed running. This seems to have fixed the issue.