topic Re: Splunkforwarder Startup Error in Docker containers in Getting Data In

Splunkforwarder Startup Error in Docker containers

samuel-devops — Wed, 05 Mar 2025 09:38:28 GMT

Commands used to run docker image: docker run -d -p 9997:9997 -p 8080:8080 -p 8089:8089 -e "SPLUNK_START_ARGS=--accept-license" -e "SPLUNK_PASSWORD=test12345" --name uf splunk/universalforwarder:latest

Seeing below error when Splunkforwarder image in starting up in docker.

2025-03-05 14:47:58 included: /opt/ansible/roles/splunk_universal_forwarder/tasks/../../../roles/splunk_common/tasks/check_for_required_restarts.yml for localhost 2025-03-05 14:47:58 Wednesday 05 March 2025 09:17:58 +0000 (0:00:00.044) 0:00:30.316 ******* 2025-03-05 14:48:31 FAILED - RETRYING: [localhost]: Check for required restarts (5 retries left). 2025-03-05 14:48:31 FAILED - RETRYING: [localhost]: Check for required restarts (4 retries left). 2025-03-05 14:48:31 FAILED - RETRYING: [localhost]: Check for required restarts (3 retries left). 2025-03-05 14:48:31 FAILED - RETRYING: [localhost]: Check for required restarts (2 retries left). 2025-03-05 14:48:31 FAILED - RETRYING: [localhost]: Check for required restarts (1 retries left). 2025-03-05 14:48:31 2025-03-05 14:48:31 TASK [splunk_universal_forwarder : Check for required restarts] **************** 2025-03-05 14:48:31 fatal: [localhost]: FAILED! => { 2025-03-05 14:48:31 "attempts": 5, 2025-03-05 14:48:31 "changed": false, 2025-03-05 14:48:31 "changed_when_result": "The conditional check 'restart_required.status == 200' failed. The error was: error while evaluating conditional (restart_required.status == 200): 'dict object' has no attribute 'status'. 'dict object' has no attribute 'status'" 2025-03-05 14:48:31 } 2025-03-05 14:48:31 2025-03-05 14:48:31 MSG: 2025-03-05 14:48:31 2025-03-05 14:48:31 GET/services/messages/restart_required?output_mode=jsonadmin********8089NoneNoneNone[200, 404];;; failed with NO RESPONSE and EXCEP_STR as Not supported URL scheme http+unix

Splunk.d is running fine, the ports are open as well Tried to curl http://localhost:8089/services/messages/restart_required?output_mode=json

Re: Splunkforwarder Startup Error in Docker containers

kiran_panchavat — Wed, 05 Mar 2025 16:01:58 GMT

@samuel-devops

Check if Splunk UF is actually running

docker ps -a | grep uf

If it’s not running, check the logs:

docker logs uf

Look for messages indicating that splunkd started and is listening on port 8089. You should see something like:

Splunk> Be an IT superhero. Splunk Universal Forwarder has started.

Confirm the ports are mapped and accessible:

docker ps

Ensure the container uf is running and ports 0.0.0.0:9997->9997/tcp, 0.0.0.0:8080->8080/tcp, and 0.0.0.0:8089->8089/tcp are listed.

Re: Splunkforwarder Startup Error in Docker containers

kiran_panchavat — Wed, 05 Mar 2025 16:04:30 GMT

@samuel-devops

Sometimes, Splunk UF fails to start due to permission issues. Ensure that the container has the right permissions:

docker exec -it uf bash
chown -R splunk:splunk /opt/splunkforwarder
chmod -R 755 /opt/splunkforwarder

Restart the container:

docker restart uf

Manually Check Splunk UF API

The error suggests that the Ansible task is failing to check for restarts via the Splunk API. Run this manually inside the container:

curl -k -u admin:test12345 https://localhost:8089/services/messages/restart_required?output_mode=json

If the API is unreachable, Splunk UF might not be fully initialized.

Re: Splunkforwarder Startup Error in Docker containers

kiran_panchavat — Wed, 05 Mar 2025 16:05:35 GMT

@samuel-devops

Make sure nothing else is using the same ports. Check if the container is binding properly:

netstat -tulnp | grep 8089

or inside the container: docker exec -it uf netstat -tulnp

Re: Splunkforwarder Startup Error in Docker containers

lackey — Wed, 05 Mar 2025 20:07:46 GMT

I think you missed the part @kiran_panchavat where @samuel-devops said splunk is up running fine.

For what it's worth, I've experienced the same thing with tags latest/9.4, 9.3, and 9.2. That last task (check_for_required_restarts) fails, but everything seems to start up fine. I will point out that this is new behavior. tag 9.3.2 for example is 4 months old and finishes it's ansible "init" as expected.

Re: Splunkforwarder Startup Error in Docker containers

lackey — Wed, 05 Mar 2025 20:36:42 GMT

It looks like the container is continually restarting. It fails that last task, aborts, and the container restarts. Splunk UF does start up as I see logs from the container in my lab's _internal index.

This looks to have changed ~7 days ago. This appears to be a broken image. Tags latest, 9.4, 9.3, 9.2, etc. Tag 9.3.2 from 4 months ago works as expected.

https://hub.docker.com/r/splunk/universalforwarder/tags

Re: Splunkforwarder Startup Error in Docker containers

lackey — Wed, 05 Mar 2025 21:19:23 GMT

To clarify, my container has a restart policy of "unless-stopped", so when the container exits after the failed ansible task, docker is restarting it. If you run it without that policy, it will run one, fail the ansible task, and exit.

Re: Splunkforwarder Startup Error in Docker containers

samuel-devops — Fri, 07 Mar 2025 10:20:37 GMT

Seems like 9.3.2 version is running fine, but still not able to send logs to splunk server which is running on EC2 instance.. below is my splunkforwarder.yml.

Can you help me with this?
It seems like forward server and monitor is not setup in the pod with below yml. How should I configure the inputs / outputs.conf files when using splunkforwarder image?

I don't see issue from splunk-server.

apiVersion: v1
kind: Pod
metadata:
name: splunk-forwarder
spec:
containers:
- name: splunk
image: splunk/universalforwarder:9.3.2
env:
- name: SPLUNK_START_ARGS
value: "--accept-license"
- name: SPLUNK_USER
value: "root"
- name: SPLUNK_PASSWORD
value: "YourSplunkPassword"
- name: SPLUNK_ADD
value: "monitor /var/logs"
- name: SPLUNK_SERVER
value: "splunk-server:9997"
volumeMounts:
- name: log-storage
mountPath: /var/logs
volumes:
- name: log-storage
persistentVolumeClaim:
claimName: log-pvc

Re: Splunkforwarder Startup Error in Docker containers

andreaska — Fri, 07 Mar 2025 12:50:06 GMT

This does not work for me

Re: Splunkforwarder Startup Error in Docker containers

andreaska — Fri, 07 Mar 2025 13:50:37 GMT

By enabling splunk_http_enabled it worked

Re: Splunkforwarder Startup Error in Docker containers

samuel-devops — Sat, 08 Mar 2025 06:52:36 GMT

Can you elaborate further, how to enable?

Re: Splunkforwarder Startup Error in Docker containers

bhupalbobbadi — Tue, 08 Apr 2025 02:49:08 GMT

I tried 9.3.2 also, it is throwing same error as described in the original post. any more steps/details how to fix this please? thanks in advance.

Re: Splunkforwarder Startup Error in Docker containers

PickleRick — Tue, 08 Apr 2025 06:02:33 GMT

Yes, the latest images seem to be broken. Apparently someone missed the fact that the UF doesn't listen on web port anymore and so the ansible task checking for open port fails.

AFAIK it has been flagged as a bug and hopefully will be resolved soon.

Re: Splunkforwarder Startup Error in Docker containers

lackey — Tue, 08 Apr 2025 16:02:30 GMT

I worked with Splunk Support and it turns out there is a known issue (a regression with one of their python libs). You can work around this by setting the environment variable ENABLE_TCP_MODE to true either at the docker run command line (-e ENABLE_TCP_MODE=true) or in your compose file (be sure if using list context to leave true unquoted).

Re: Splunkforwarder Startup Error in Docker containers

PickleRick — Tue, 08 Apr 2025 16:10:18 GMT

Hmm... There is not a single mention of this variable in https://splunk.github.io/splunk-ansible/ADVANCED.html#inventory-script

What is it supposed to do?

Re: Splunkforwarder Startup Error in Docker containers

sylim_splunk — Fri, 11 Apr 2025 00:18:18 GMT

It appears hitting a known issue for some recent versions below.

9.4.1
9.4.0
9.3.3
9.2.5
9.1.8

You may want to check this article.

https://github.com/splunk/docker-splunk/issues/698