As a POC, I have been able to get the Splunk forwarder working but it requires a lot of tweaking (def. not enterprise ready).
Basically I followed this guide: https://docs.splunk.com/Documentation/Forwarder/7.2.6/Forwarder/Makeauniversalforwarderpartofahostimage
In order for this to work, here are the things I did:
On a separate machine with Docker (not OpenShift), create a Dockerfile or use the one on github: https://github.com/splunk/docker-splunk/blob/develop/splunk/common-files/Dockerfile
Get it working, then run the command to clear clone-prep-clear-config command: ./splunk clone-prep-clear-config
Use the docker cp command to copy the etc, var, and share from $SPLUNK_HOME
Zip those directories.
In OpenShift, add persistent storage for the etc, var and share paths as volumes
In my entrypoint.sh file (Just did it here as testing), I added some logic to populate etc, var and share paths with the zips created in step 4.
Start service.
Note: I running this as a side-car solution where I have the app in the same project. The app is writing logs to another persistent volume that is shared with the splunk-forwarder pod and splunk-forwarder is configured to read from that persistent volume.
Going forward we're going to look at fluentd forwarding logs to splunk.
... View more