Splunkforwarder Startup Error in Docker containers

[samuel-devops]

Commands used to run docker image: docker run -d -p 9997:9997 -p 8080:8080 -p 8089:8089 -e "SPLUNK_START_ARGS=--accept-license" -e "SPLUNK_PASSWORD=test12345" --name uf splunk/universalforwarder:latest

Seeing below error when Splunkforwarder image in starting up in docker.

2025-03-05 14:47:58 included: /opt/ansible/roles/splunk_universal_forwarder/tasks/../../../roles/splunk_common/tasks/check_for_required_restarts.yml for localhost 2025-03-05 14:47:58 Wednesday 05 March 2025 09:17:58 +0000 (0:00:00.044) 0:00:30.316 ******* 2025-03-05 14:48:31 FAILED - RETRYING: [localhost]: Check for required restarts (5 retries left). 2025-03-05 14:48:31 FAILED - RETRYING: [localhost]: Check for required restarts (4 retries left). 2025-03-05 14:48:31 FAILED - RETRYING: [localhost]: Check for required restarts (3 retries left). 2025-03-05 14:48:31 FAILED - RETRYING: [localhost]: Check for required restarts (2 retries left). 2025-03-05 14:48:31 FAILED - RETRYING: [localhost]: Check for required restarts (1 retries left). 2025-03-05 14:48:31 2025-03-05 14:48:31 TASK [splunk_universal_forwarder : Check for required restarts] **************** 2025-03-05 14:48:31 fatal: [localhost]: FAILED! => { 2025-03-05 14:48:31 "attempts": 5, 2025-03-05 14:48:31 "changed": false, 2025-03-05 14:48:31 "changed_when_result": "The conditional check 'restart_required.status == 200' failed. The error was: error while evaluating conditional (restart_required.status == 200): 'dict object' has no attribute 'status'. 'dict object' has no attribute 'status'" 2025-03-05 14:48:31 } 2025-03-05 14:48:31 2025-03-05 14:48:31 MSG: 2025-03-05 14:48:31 2025-03-05 14:48:31 GET/services/messages/restart_required?output_mode=jsonadmin********8089NoneNoneNone[200, 404];;; failed with NO RESPONSE and EXCEP_STR as Not supported URL scheme http+unix

Splunk.d is running fine, the ports are open as well Tried to curl http://localhost:8089/services/messages/restart_required?output_mode=json

[lackey]

To clarify, my container has a restart policy of "unless-stopped", so when the container exits after the failed ansible task, docker is restarting it. If you run it without that policy, it will run one, fail the ansible task, and exit.

[lackey · Accepted solution]

It looks like the container is continually restarting. It fails that last task, aborts, and the container restarts. Splunk UF does start up as I see logs from the container in my lab's _internal index.

This looks to have changed ~7 days ago. This appears to be a broken image. Tags latest, 9.4, 9.3, 9.2, etc. Tag 9.3.2 from 4 months ago works as expected.

https://hub.docker.com/r/splunk/universalforwarder/tags

[PickleRick]

Yes, the latest images seem to be broken. Apparently someone missed the fact that the UF doesn't listen on web port anymore and so the ansible task checking for open port fails.

AFAIK it has been flagged as a bug and hopefully will be resolved soon.

[lackey]

I worked with Splunk Support and it turns out there is a known issue (a regression with one of their python libs). You can work around this by setting the environment variable ENABLE_TCP_MODE to true either at the docker run command line (-e ENABLE_TCP_MODE=true) or in your compose file (be sure if using list context to leave true unquoted).

[PickleRick]

Hmm... There is not a single mention of this variable in https://splunk.github.io/splunk-ansible/ADVANCED.html#inventory-script

What is it supposed to do?

[bhupalbobbadi]

I tried 9.3.2 also, it is throwing same error as described in the original post. any more steps/details how to fix this please? thanks in advance.

[samuel-devops]

Seems like 9.3.2 version is running fine, but still not able to send logs to splunk server which is running on EC2 instance.. below is my splunkforwarder.yml.

Can you help me with this?
It seems like forward server and monitor is not setup in the pod with below yml. How should I configure the inputs / outputs.conf files when using splunkforwarder image?

I don't see issue from splunk-server.

apiVersion: v1
kind: Pod
metadata:
name: splunk-forwarder
spec:
containers:
- name: splunk
image: splunk/universalforwarder:9.3.2
env:
- name: SPLUNK_START_ARGS
value: "--accept-license"
- name: SPLUNK_USER
value: "root"
- name: SPLUNK_PASSWORD
value: "YourSplunkPassword"
- name: SPLUNK_ADD
value: "monitor /var/logs"
- name: SPLUNK_SERVER
value: "splunk-server:9997"
volumeMounts:
- name: log-storage
mountPath: /var/logs
volumes:
- name: log-storage
persistentVolumeClaim:
claimName: log-pvc

[andreaska]

This does not work for me

[andreaska]

By enabling splunk_http_enabled it worked

[samuel-devops]

Can you elaborate further, how to enable?

[lackey]

I think you missed the part @kiran_panchavat where @samuel-devops said splunk is up running fine.

For what it's worth, I've experienced the same thing with tags latest/9.4, 9.3, and 9.2. That last task (check_for_required_restarts) fails, but everything seems to start up fine. I will point out that this is new behavior. tag 9.3.2 for example is 4 months old and finishes it's ansible "init" as expected.

[kiran_panchavat]

@samuel-devops 

Make sure nothing else is using the same ports. Check if the container is binding properly:

netstat -tulnp | grep 8089

or inside the container: docker exec -it uf netstat -tulnp

[kiran_panchavat]

@samuel-devops 

Sometimes, Splunk UF fails to start due to permission issues. Ensure that the container has the right permissions:

docker exec -it uf bash
chown -R splunk:splunk /opt/splunkforwarder
chmod -R 755 /opt/splunkforwarder

Restart the container:

docker restart uf

 Manually Check Splunk UF API

The error suggests that the Ansible task is failing to check for restarts via the Splunk API. Run this manually inside the container:

curl -k -u admin:test12345 https://localhost:8089/services/messages/restart_required?output_mode=json

If the API is unreachable, Splunk UF might not be fully initialized.

 

[kiran_panchavat]

@samuel-devops 

Check if Splunk UF is actually running 

docker ps -a | grep uf

If it’s not running, check the logs:

docker logs uf

Look for messages indicating that splunkd started and is listening on port 8089. You should see something like:

Splunk> Be an IT superhero. Splunk Universal Forwarder has started.

Confirm the ports are mapped and accessible:

docker ps

 Ensure the container uf is running and ports 0.0.0.0:9997->9997/tcp, 0.0.0.0:8080->8080/tcp, and 0.0.0.0:8089->8089/tcp are listed.

[sylim_splunk]

It appears hitting a known issue for some recent versions below.

• 9.4.1
• 9.4.0
• 9.3.3
• 9.2.5
• 9.1.8

You may want to check this article.

https://github.com/splunk/docker-splunk/issues/698