Deploying Splunk Universal Forwarder with Ansible gets stuck on start command

[magguevara]

Hi All,

I have been working on a ansible playbook to deploy the UF to different servers. So far everything is working fine until the playbook tries to execute the command to start splunk the first time

Code is as follows

- name: Start splunk service
  become: true
  become_method: sudo
  become_user: splunk
  command: /opt/splunkforwarder/bin/splunk start --accept-license --answer-yes --no-prompt --seed-passwd {{uf_user_password}}
  register: console

Ansible just gets stuck there and task doesn't end, if you check the server you can see that the command executed is the correct one even with the right user but nothing happens

If you run the command with the same user on the server we get this

Warning: Attempting to revert the SPLUNK_HOME ownership
Warning: Executing "chown -R splunk /opt/splunkforwarder"

This appears to be your first time running this version of Splunk.
Creating unit file...
Error calling execve(): No such file or directory
Error launching command: No such file or directory
Failed to create the unit file. Please do it manually later.


Splunk> The Notorious B.I.G. D.A.T.A.

Checking prerequisites...
Checking mgmt port [8089]: open
Creating: /opt/splunkforwarder/var/lib/splunk
Creating: /opt/splunkforwarder/var/run/splunk
Creating: /opt/splunkforwarder/var/run/splunk/appserver/i18n
Creating: /opt/splunkforwarder/var/run/splunk/appserver/modules/static/css
Creating: /opt/splunkforwarder/var/run/splunk/upload
Creating: /opt/splunkforwarder/var/run/splunk/search_telemetry
Creating: /opt/splunkforwarder/var/spool/splunk
Creating: /opt/splunkforwarder/var/spool/dirmoncache
Creating: /opt/splunkforwarder/var/lib/splunk/authDb
Creating: /opt/splunkforwarder/var/lib/splunk/hashDb
New certs have been generated in '/opt/splunkforwarder/etc/auth'.
Checking conf files for problems...
Invalid key in stanza [webhook] in /opt/splunkforwarder/etc/system/default/alert_actions.conf, line 229: enable_allowlist (value: false).
Your indexes and inputs configurations are not internally consistent. For more information, run 'splunk btool check --debug'
Done
Checking default conf files for edits...
Validating installed files against hashes from '/opt/splunkforwarder/splunkforwarder-9.0.4-de405f4a7979-linux-2.6-x86_64-manifest'
All installed files intact.
Done
All preliminary checks passed.

Starting splunk server daemon (splunkd)... 
PYTHONHTTPSVERIFY is set to 0 in splunk-launch.conf disabling certificate validation for the httplib and urllib libraries shipped with the embedded Python interpreter; must be set to "1" for increased security
Done
[ OK ]

We have also tried different approaches like using a script file and executing it instead of calling directly the command, but always get the same results. 

Any suggestions?

Regards

[PickleRick]

Increase your ansible verbosity and see what's happening "underneath".

[magguevara]

Already tried that with up to 4 levels of verbose and always happens the same once it reaches the task that executes the command it gets stuck there.

'/bin/sh -c '"'"'sudo -H -S -n -u splunk /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-rwkwgkjbkdrirllfwqefbvoxzvjuexbf ; /usr/bin/python /var/tmp/ansible-tmp-1686172641.9378927-7562-175490366307620/AnsiballZ_command.py'"'"'"'"'"'"'"'"' && sleep 0'"'"''
Escalation succeeded.

One important thing is that other commands like the one below complete just fine. It is the splunk start with license acceptance that fails to continue

- name: Disable expiration
  shell: chage -E -1 -M -1 -W -1 -m -1 splunk

[isoutamo · Accepted solution]

Hi

my guess is that is somehow still waiting input on command prompt? 

I usually use user-seed.conf file with crypted password. 

- name: Set admin access via seed
  when: splunk_first_run | bool
  block:

    - name: "Hash the password"
      command: "{{ splunk.exec }} hash-passwd {{ splunk.password }}"
      register: hashed_pwd
      changed_when: hashed_pwd.rc == 0
      become: yes
      become_user: "{{ splunk.user }}"
      no_log: "{{ hide_password }}"

    - name: "Generate user-seed.conf (Linux)"
      ini_file:
        owner: "{{ splunk.user }}"
        group: "{{ splunk.group }}"
        dest: "{{ splunk.home }}/etc/system/local/user-seed.conf"
        section: user_info
        option: "{{ item.opt }}"
        value: "{{ item.val }}"
        mode: 0644
      with_items:
        - {opt: 'USERNAME', val: '{{ splunk.admin_user }}'}
        - {opt: 'HASHED_PASSWORD', val: '{{ hashed_pwd.stdout }}'}
      loop_control:
        label: "{{ item.opt }}"
      when: ansible_system is match("Linux")
      become: yes
      become_user: "{{ splunk.user }}"
      no_log: "{{ hide_password }}"

Quite much similarities with splunk-ansible 😉

r. Ismo