Struggling with universal forwarder docker container

[stringa]

I had this working at some point, but I am not able to get any of the commands to run after the universal forwarder starts up. At one point, I was able to see it add directories, but now I'm not sure what is wrong with my configuration that doesn't allow it to start run the startup commands

  vsplunk_uf:
    container_name: vsplunk_uf
    image: busybox
    volumes:
      - splunk-etc:/opt/splunk/etc
      - splunk-var:/opt/splunk/var

  splunkuniversalforwarder:
    image: splunk/universalforwarder:latest
    hostname: splunkuniversalforwarder
    environment:
       SPLUNK_START_ARGS: --accept-license --answer-yes
       SPLUNK_USER: root
       SPLUNK_CMD: 'add monitor -source /usr/local/localrw/logs/webrtc/* -index webrtc -sourcetype signaler'
    volumes:
      - /var/lib/docker/containers:/host/containers:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - splunk-etc:/opt/splunk/etc
      - splunk-var:/opt/splunk/var
      - signaler-logs:/usr/local/localrw/logs/webrtc
      - kms-logs:/var/log/kurento-media-server/
    depends_on:
      - webrtc.server.1
      - kms.1
      - vsplunk_uf

I am seeing the server logs stop here:

splunkuniversalforwarder_1  |   Checking conf files for problems...
splunkuniversalforwarder_1  |   Done
splunkuniversalforwarder_1  |   Checking default conf files for edits...
splunkuniversalforwarder_1  |   Validating installed files against hashes from '/opt/splunk/splunkforwarder-6.5.3-36937ad027d4-linux-2.6-x86_64-manifest'
splunkuniversalforwarder_1  |   All installed files intact.
splunkuniversalforwarder_1  |   Done
splunkuniversalforwarder_1  | All preliminary checks passed.
splunkuniversalforwarder_1  | 
splunkuniversalforwarder_1  | Starting splunk server daemon (splunkd)...  
splunkuniversalforwarder_1  | Done
splunkuniversalforwarder_1  | 

And when I try to run: docker exec webrtcserver_splunkuniversalforwarder_1 entrypoint.sh splunk list monitor

Authentication needed, run "splunk login"

I've seen this run successfully, but I'm not sure what I'm doing wrong at this point and why it doesn't seem to be running the commands setup in the environment variables.

Thanks for any tips

[stringa · Accepted solution]

It ended up being a problem with docker sharing volumes. As I was running all my tests, the volumes were not going away and just caused some problems.

I ended up aborting on trying to get the Universal Forwarder working in our environment since I would have to build a new custom docker container to handle setting up proper SSL certs, and I've moved to using a Splunk docker log driver.

[koshyk]

I won't do inputs as SPLUNK_CMD, but rather I will put them into an app (eg my_collection_app).
then I would do

docker cp <your_host> <uf_container_etc_apps> # also permissions etc.

(or manage using deployment-server)

This way you can retain the purity of your docker image, but all other configurations managed from a central Repository location.

[stringa]

What I am noticing is that I get the migration warning and then configuration doesn't seem to set. I'm literally using the docker container, but I've had to convert it to a docker-compose v3. I'm able to get that working independately, but when integrating it into my version, it seems to fail.

[koshyk]

hmm..very specific.
do you use just dockerfile or docker-compose? I used docker-compose and was very smooth

[stringa]

I've noticed that I am getting a migration warning in my configuration and when I run a stripped down example, I am not getting that warning.

What I am seeing in mine:

This appears to be an upgrade of Splunk.

I imagine if I can get rid of this warning it will work.