Splunk Enterprise

Docker-splunk failing "Check Splunk instance is running" ansible check, but splunk is indeed running

scoady
New Member

Hi everyone! Hoping I might be missing something simple.

 

We're running splunk enterprise 8.1.0 with the officially distributed docker image. All is well with our search head cluster, with one slightly difficult-to-track-down issue that has been causing frequent restarts of our search head tasks.

 

 

Everything starts up cleanly, we have a good search head cluster, UIs are returning results normally, etc. But, it appears that an ansible health check that runs at the very end of the playbook is failing to validate that the splunkweb UI is up and running (it is).

 

 

included: /opt/ansible/roles/splunk_search_head/tasks/../../../roles/splunk_common/tasks/wait_for_splunk_instance.yml for localhost
Monday 23 August 2021 23:48:24 +0000 (0:00:00.045) 0:00:59.426 *********
FAILED - RETRYING: Check Splunk instance is running (60 retries left).

 

 

This will eventually fail after 60 retries and will force the container to restart, briefly disrupting the search head cluster.

 

 

We haven't overridden many options on the web.conf side aside from setting up ProxySSO (this was happening before configuring SSO also).

 

According to the file, this is the configured check:

---
- name: Check Splunk instance is running
uri:
url: "{{ scheme | default(cert_prefix) }}://{{ splunk_instance_address }}:{{ port | default(splunk.svc_port) }}"
method: GET
validate_certs: false
use_proxy: no
register: task_response
until:
- task_response.status == 200
retries: "{{ wait_for_splunk_retry_num }}"
delay: "{{ retry_delay }}"
ignore_errors: true
no_log: "{{ hide_password }}"

 

I can't see anything in the ansible logs detailing what that URL renders as.

 

My suspicion is that this is attempting to contact either the wrong hostname, is using SSL (we have disabled SSL and are terminating on a reverse proxy), but I can't find any evidence to back that up.

 

Is there something else I can do to force that check to use http://localhost:8000?

 

 

Thanks!!

Labels (4)
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...