Splunk Docker: committing container as new image

[breid1313]

Hi all,

I'm leveraging the splunk/splunk:8.0.4.1 image for some development. At this time, I've pulled the splunk/splunk image, and configured it according to my needs. I now want to commit that container to save it as a new image, so that I have a baseline container to fall back on and start. However, my newly saved image fails to start when I perform a `docker run`. I've checked the original container for the `cacert.pem` file in question and it does exist. Since the new image I'm attempting to start is essentially a clone of the running container, I'm surprised that it fails to start.

Is this a use case that can be achieved with the splunk/splunk image? Thank you.

```
TASK [splunk_common : Start Splunk via CLI] ************************************
fatal: [localhost]: FAILED! => {
"changed": false,
"cmd": [
"/opt/splunk/bin/splunk",
"start",
"--accept-license",
"--answer-yes",
"--no-prompt"
],
"delta": "0:00:01.506092",
"end": "2020-07-15 19:59:26.957025",
"rc": 1,
"start": "2020-07-15 19:59:25.450933"
}

STDOUT:


Splunk> Take the sh out of IT.

Checking prerequisites...
Checking http port [8000]: open
Checking mgmt port [8089]: open
Checking appserver port [127.0.0.1:8065]: open
Checking kvstore port [8191]: open
Checking configuration... Done.
Creating: /opt/splunk/var/lib/splunk
Creating: /opt/splunk/var/run/splunk
Creating: /opt/splunk/var/run/splunk/appserver/i18n
Creating: /opt/splunk/var/run/splunk/appserver/modules/static/css
Creating: /opt/splunk/var/run/splunk/upload
Creating: /opt/splunk/var/run/splunk/search_telemetry
Creating: /opt/splunk/var/spool/splunk
Creating: /opt/splunk/var/spool/dirmoncache
Creating: /opt/splunk/var/lib/splunk/authDb
Creating: /opt/splunk/var/lib/splunk/hashDb


STDERR:

The CA file specified (/opt/splunk/etc/auth/cacert.pem) does not exist. Cannot continue.
SSL certificate generation failed.


MSG:

non-zero return code

PLAY RECAP *********************************************************************
localhost : ok=43 changed=6 unreachable=0 failed=1 skipped=44 rescued=0 ignored=0

```

[tuckcodes]

Having the same issue. This makes it very difficult to leverage modern tools like Gitlab, where having a base image is important. Furthermore, "-e SPLUNK_APPS_URL=<url>" does not seam to work either in the docker run command. These are very common requirement for modern deployment via CI/CD, regardless of the platform..

• There is no way to build an image and automate the installation of apps via '-e' in the docker run command.
• There is also no way to build the vanilla splunk/spunk:latest image, install the apps manually via localhost, and then commit the container state to a new image. 

It appears as though deploying Splunk via containers is not yet a viable option. Please let me know if I am missing something, as deploying Splunk via containers (rather than a VM) is my preferred path.

[breid1313 · Accepted solution]

@tuckcodes have a look at this. Hope it helps!

https://github.com/breid1313/splunk-docker

[breid1313]

@tuckcodes I have something hacky working that I can generalize and push to github if you're interested. It's a docker-compose setup that creates a new slave server off a master license server we have running remotely. It's pretty convenient for quick testing or to more closely replicate someone else's environment.

[breid1313]

I have also tried to copy /opt/splunk/etc/ and /opt/splunk/var/ to my local file system and mount them into a new container with the below command:

docker run -it \
-p 8000:8000 \
-e "SPLUNK_START_ARGS=--accept-license" \
-e "SPLUNK_PASSWORD=securePassw0rd" \
-v /local/path/etc/:/opt/splunk/etc \
-v /local/path/var/:/opt/splunk/var \
splunk/splunk

but ansible still fails to provision the new server

TASK [splunk_common : Start Splunk via CLI] *********************************************************************************************************************************************************************************************************************************
fatal: [localhost]: FAILED! => {
"changed": false,
"cmd": [
"/opt/splunk/bin/splunk",
"start",
"--accept-license",
"--answer-yes",
"--no-prompt"
],
"delta": "0:00:16.962175",
"end": "2020-07-15 22:28:05.187999",
"rc": 10,
"start": "2020-07-15 22:27:48.225824"
}

STDOUT:


Splunk> Take the sh out of IT.

Checking prerequisites...
Checking http port [8000]: open
Checking mgmt port [8089]: open
Checking appserver port [127.0.0.1:8065]: open
Checking kvstore port [8191]: open
Checking configuration... Done.
Checking critical directories... Done
Checking indexes...


STDERR:

homePath='/opt/splunk/var/lib/splunk/audit/db' of index=_audit on unusable filesystem.
Validating databases (splunkd validatedb) failed with code '1'. If you cannot resolve the issue(s) above after consulting documentation, please file a case online at http://www.splunk.com/page/submit_issue


MSG:

non-zero return code