Why am I unable to start splunk for first time?

[bobmcperson]

Hello there,

I am attempting to install splunk forwarder version 8.2.3-cd0848707637 on a RHEL 7.8 server using an ansible script.  Here is the ansible script:

- hosts: all 
  vars_prompt:
    - name: splunk_idxcluster 
      prompt: What is the indexer password for symmetric key?

    - name: splunk_uf_admin 
      prompt: What is the admin password?

  vars:
    deploymentserver: ***************************
    clustermaster: ****************************

  tasks:
    - name: Install Splunk UF package
      become: yes
      become_method: dzdo
      yum:
        name: splunkforwarder
        state: latest
      register: splunkpackage

    - name: Check if ftr file exists
      stat:
        path: /opt/splunkforwarder/ansible_splunk.ftr
      register: splunkftr

    - name: Set ACL to allow splunk user access to log files
      become: yes
      become_method: dzdo
      acl:
        path: "/opt/splunkforwarder"
        recursive: yes 
        permissions: rwx
        entity: splunk
        etype: group
        state: present

    - name: Create user-seed file
      become: yes
      become_method: dzdo
      copy:
        dest: "/opt/splunkforwarder/etc/system/local/user-seed.conf"
        owner: splunk
        group: splunk
        content: |
              [user_info]
              USERNAME = admin
              PASSWORD = {{ splunk_uf_admin }}
      register: seedfile
      when: splunkftr.stat.exists == False
    
    - name: First run to accept license
      become_user: splunk
      become_method: dzdo
      command: /opt/splunkforwarder/bin/splunk start --accept-license --no-prompt
      register: splunklicense
      when: splunkftr.stat.exists == False

    - name: Enable UF to start at boot as user splunk
      shell: /opt/splunkforwarder/bin/splunk enable boot-start -user splunk
      register: splunkbootstart
      when: splunkftr.stat.exists == False

    - name: Build outputs.conf to check-in with indexer cluster master
      copy:
        dest: "/opt/splunkforwarder/etc/system/local/outputs.conf"
        owner: splunk
        group: splunk
        content: |
              [tcpout]
              defaultGroup = cluster
              [tcpout:cluster]
              indexerDiscovery = cluster
              clientCert = /opt/splunkforwarder/etc/auth/server.pem
              sslPassword = password
              sslRootCAPath = /opt/splunkforwarder/etc/auth/cacert.pem
              sslVerifyServerCert = false
              useACK = true
              [indexer_discovery:cluster]
              pass4SymmKey = {{ splunk_idxcluster }}
              master_uri = {{ clustermaster }}
      when: splunkftr.stat.exists == False

    - name: Configure UF to check-in with the deployment server
      shell: "sudo -u splunk /opt/splunkforwarder/bin/splunk set deploy-poll {{ deploymentserver}} -auth admin:{{ splunk_uf_admin }}"
      when: splunkftr.stat.exists == False

    - name: Create Splunk ftr file to convey Splunk has already been setup before
      file:
        path: "/opt/splunkforwarder/ansible_splunk.ftr"
        state: touch
      when: splunklicense is succeeded and splunkbootstart is succeeded

    - name: Set ACL to allow splunk user access to log files
      acl:
        path: "/u/log"
        recursive: yes 
        permissions: rwx
        entity: splunk
        etype: group
        state: present

    - name: Add /u/log to watched directories
      shell: /opt/splunkforwarder/bin/splunk add monitor /u/log -index applogs -sourcetype %elasticsearch%
      ignore_errors: yes

When I get to the "First run to accept license" step, I get an error "Please run 'splunk ftr' as boot-start user".  I have tried running "splunk ftr" as splunk, but then I get the same error, so I'm not sure it's doing anything.  Any help would be greatly appreciated!

[hocuspocus]

May need to configure the "boot-start" user. Here is the link:

https://docs.splunk.com/Documentation/Splunk/8.2.4/Admin/ConfigureSplunktostartatboottime#Enable_boo...

[bobmcperson]

I followed the "Enable boot-start on machines that run systemd" instructions, and I'm getting this error on the step to start splunk:  "Failed to run splunk as SPLUNK_OS_USER. This command can only be run by bootstart user."

Here's my modified ansible script:

- hosts: all 
  vars_prompt:
    - name: splunk_idxcluster 
      prompt: What is the indexer password for symmetric key?

    - name: splunk_uf_admin 
      prompt: What is the admin password?

  vars:
    deploymentserver: **********************
    clustermaster: *******************

  tasks:
    - name: Install Splunk UF package
      become: yes
      become_method: dzdo
      yum:
        name: splunkforwarder
        state: latest
      register: splunkpackage

    - name: Check if ftr file exists
      stat:
        path: /opt/splunkforwarder/ansible_splunk.ftr
      register: splunkftr

    - name: Create user-seed file
      become: yes
      become_method: dzdo
      copy:
        dest: "/opt/splunkforwarder/etc/system/local/user-seed.conf"
        owner: splunk
        group: splunk
        content: |
              [user_info]
              USERNAME = admin
              PASSWORD = {{ splunk_uf_admin }}
      register: seedfile
      when: splunkftr.stat.exists == False

    - name: Disable init.d bootstart service
      become: yes
      become_method: dzdo
      shell: /opt/splunkforwarder/bin/splunk disable boot-start
      ignore_errors: yes

    - name: Enable UF to start at boot as user splunk
      become: yes
      become_method: dzdo
      shell: /opt/splunkforwarder/bin/splunk enable boot-start -user splunk -systemd-managed 1
      register: splunkbootstart
      when: splunkftr.stat.exists == False

    - name: Make splunk user owner of $SPLUNK_HOME
      become: yes
      become_method: dzdo
      ansible.builtin.file:
        path: "/opt/splunkforwarder"
        recurse: yes 
        owner: splunk
        group: splunk

    - name: First run to accept license
      become_user: splunk 
      become_method: dzdo
      command: /opt/splunkforwarder/bin/splunk start --accept-license --no-prompt
      register: splunklicense
      when: splunkftr.stat.exists == False

[hocuspocus]

Seems as though the bootstart user cannot run the service? Here is a link for running splunk as a systemd service:

https://docs.splunk.com/Documentation/Splunk/8.2.4/Admin/RunSplunkassystemdservice