Why is universal forwarder not starting on docker image >= 9.0?

[jplassnibatt]

Hello everyone.

First of all, this was working fine using images 8.x.

Here is my compose for 8.2:

version: '3.6'
services:
  splunkuf82:
    tty: true
    image: splunk/universalforwarder:8.2
    hostname: universalforwarder82
    container_name: universalforwarder82
    environment:
      SPLUNK_START_ARGS: "--accept-license --answer-yes --no-prompt"
      SPLUNK_USER: root
      SPLUNK_GROUP: root
      SPLUNK_PASSWORD: "adminadmin"

Here are some commands to check if it is running:

jpla@rd:~/pd/rd/docker/rundeck/rd.universalforwarder82$ docker compose down
jpla@rd:~/pd/rd/docker/rundeck/rd.universalforwarder82$ docker compose up -d
[+] Running 2/2
 ⠿ Network rduniversalforwarder82_default  Created   0.1s
 ⠿ Container universalforwarder82          Started   0.4s
jpla@rd:~/pd/rd/docker/rundeck/rd.universalforwarder82$ docker exec -it universalforwarder82 bash
[ansible@universalforwarder82 splunkforwarder]$ cd bin
[ansible@universalforwarder82 bin]$ sudo ./splunk status
splunkd is running (PID: 1125).
splunk helpers are running (PIDs: 1126).

Here is my compose for 9.0.3:

version: '3.6'
services:
  splunkuf903:
    tty: true
    image: splunk/universalforwarder:9.0.3
    hostname: universalforwarder903
    container_name: universalforwarder903
    environment:
      SPLUNK_START_ARGS: "--accept-license --answer-yes --no-prompt"
      SPLUNK_USER: root
      SPLUNK_GROUP: root
      SPLUNK_PASSWORD: "adminadmin"

Here are the same commands to check if it is running:

jpla@rd:~/pd/rd/docker/rundeck/rd.universalforwarder903$ docker compose down
jpla@rd:~/pd/rd/docker/rundeck/rd.universalforwarder903$ docker compose up -d
[+] Running 2/2
 ⠿ Network rduniversalforwarder903_default  Created    0.1s
 ⠿ Container universalforwarder903          Started    0.5s
jpla@rd:~/pd/rd/docker/rundeck/rd.universalforwarder903$ docker exec -it universalforwarder903 bash
[ansible@universalforwarder903 splunkforwarder]$ cd bin
[ansible@universalforwarder903 bin]$ sudo ./splunk status
Warning: Attempting to revert the SPLUNK_HOME ownership
Warning: Executing "chown -R root /opt/splunkforwarder"
Error calling execve(): No such file or directory
Error launching  command: No such file or directory
execvp: No such file or directory
Do you agree with this license? [y/n]: y

This appears to be an upgrade of Splunk.
--------------------------------------------------------------------------------)

Splunk has detected an older version of Splunk installed on this machine. To
finish upgrading to the new version, Splunk's installer will automatically
update and alter your current configuration files. Deprecated configuration
files will be renamed with a .deprecated extension.

You can choose to preview the changes that will be made to your configuration
files before proceeding with the migration and upgrade:

If you want to migrate and upgrade without previewing the changes that will be
made to your existing configuration files, choose 'y'.
If you want to see what changes will be made before you proceed with the
upgrade, choose 'n'.


Perform migration and upgrade without previewing configuration changes? [y/n] y

-- Migration information is being logged to '/opt/splunkforwarder/var/log/splunk/migration.log.2023-01-31.23-16-18' --

Migrating to:
VERSION=9.0.3
BUILD=dd0128b1f8cd
PRODUCT=splunk
PLATFORM=Linux-x86_64


Error calling execve(): No such file or directory
Error launching  command: Invalid argument
^C
[ansible@universalforwarder903 bin]$ sudo ./splunk status
Warning: Attempting to revert the SPLUNK_HOME ownership
Warning: Executing "chown -R root /opt/splunkforwarder"
Error calling execve(): No such file or directory
Error launching  command: No such file or directory
execvp: No such file or directory
Do you agree with this license? [y/n]:

As you can see in 9.0.3 it asks for license again, and again after saying yes the first time.

This behaviour is running on Docker version 20.10.23, also happening on Minikube version: v1.29.0.- on Linuxmint 21.1.-

I added tty: true per this recommendation, but it didn't work for me.

Could anybody please confirm the issue?

Thanks!

[EshwarM]

Hi

We are having similar issues with version 9.05 on Debian 11. Do we have a fix for this?

Regards

Eshwar