Upgrading Universal Forwarder 8.x.x to 9.x.x does not work?

[blaha1]

I have been using the Universal  forwarder splunkforwarder-7.2.6-c0bf0f679ce9-Linux-x86_64 for quite a while without issues. I now wanted to upgrade to the latest one, 9.0.2 so I downloaded it and ran it just like I did with the old version. However, when starting it, 

${SPLUNK_HOME}/bin/splunk start --accept-license --answer-yes --no-prompt

It seems to crash with

Error calling execve(): No such file or directory
Error launching command: Invalid argument

I then tried the latest 8.x.x version, 8.2.9 and that worked perfectly fine.

What has changed between version 8 and 9? Any new requirements I am not aware of?

[blaha1]

I have now done some additional research and testing.

I am using Alpine Linux which does not include systemd. That is probably why this is not working for me.

8e23f2b85b3a:/# "/opt/splunkforwarder/bin/splunk" start --accept-license --answer-yes --no-prompt
Warning: Attempting to revert the SPLUNK_HOME ownership
Warning: Executing "chown -R splunk:splunk /opt/splunkforwarder"

This appears to be your first time running this version of Splunk.
Creating unit file...
Error calling execve(): No such file or directory
Error launching command: No such file or directory
Failed to create the unit file. Please do it manually later.

Splunk> The Notorious B.I.G. D.A.T.A.

Checking prerequisites...
Checking mgmt port [8089]: open
Creating: /opt/splunkforwarder/var/lib/splunk
Creating: /opt/splunkforwarder/var/run/splunk
Creating: /opt/splunkforwarder/var/run/splunk/appserver/i18n
Creating: /opt/splunkforwarder/var/run/splunk/appserver/modules/static/css
Creating: /opt/splunkforwarder/var/run/splunk/upload
Creating: /opt/splunkforwarder/var/run/splunk/search_telemetry
Creating: /opt/splunkforwarder/var/run/splunk/search_log
Creating: /opt/splunkforwarder/var/spool/splunk
Creating: /opt/splunkforwarder/var/spool/dirmoncache
Creating: /opt/splunkforwarder/var/lib/splunk/authDb
Creating: /opt/splunkforwarder/var/lib/splunk/hashDb
Checking conf files for problems...
Done
Checking default conf files for edits...
Validating installed files against hashes from '/opt/splunkforwarder/splunkforwarder-9.1.2-b6b9c8185839-linux-2.6-x86_64-manifest'
All installed files intact.
Done
All preliminary checks passed.

Starting splunk server daemon (splunkd)...
PYTHONHTTPSVERIFY is set to 0 in splunk-launch.conf disabling certificate validation for the httplib and urllib libraries shipped with the embedded Python interpreter; must be set to "1" for increased security

However it seems to start a background process but I dont see the logs in splunk. Using the status command kills the background process:

8e23f2b85b3a:/# "/opt/splunkforwarder/bin/splunk" status
Warning: Attempting to revert the SPLUNK_HOME ownership
Warning: Executing "chown -R splunk:splunk /opt/splunkforwarder"
splunkd 165 was not running.
Stopping splunk helpers...

I have tried disabling boot start:

splunk disable boot-start

But that gives me a similar error:

Error calling execve(): No such file or directory
Error launching command: No such file or directory
execve: No such file or directory
while running command /sbin/chkconfig

Has something changed from 8.x to 9.x that now systemd is used default somehow? How can I run the universal forwarder without systemd?

[blaha1]

Still having this error with 9.0.4 I'm afraid.

50b81383ef0d:/opt/splunkforwarder/bin# ./splunk start --accept-license --answer-yes --no-prompt
Warning: Attempting to revert the SPLUNK_HOME ownership
Warning: Executing "chown -R splunk /opt/splunkforwarder"

This appears to be your first time running this version of Splunk.
Creating unit file...
Error calling execve(): No such file or directory
Error launching  command: No such file or directory
Failed to create the unit file. Please do it manually later.


Splunk> The Notorious B.I.G. D.A.T.A.

Checking prerequisites...
        Checking mgmt port [8089]: open
                Creating: /opt/splunkforwarder/var/lib/splunk
                Creating: /opt/splunkforwarder/var/run/splunk
                Creating: /opt/splunkforwarder/var/run/splunk/appserver/i18n
                Creating: /opt/splunkforwarder/var/run/splunk/appserver/modules/static/css
                Creating: /opt/splunkforwarder/var/run/splunk/upload
                Creating: /opt/splunkforwarder/var/run/splunk/search_telemetry
                Creating: /opt/splunkforwarder/var/spool/splunk
                Creating: /opt/splunkforwarder/var/spool/dirmoncache
                Creating: /opt/splunkforwarder/var/lib/splunk/authDb
                Creating: /opt/splunkforwarder/var/lib/splunk/hashDb
        Checking conf files for problems...
                Invalid key in stanza [webhook] in /opt/splunkforwarder/etc/system/default/alert_actions.conf, line 229: enable_allowlist (value: false).
                Your indexes and inputs configurations are not internally consistent. For more information, run 'splunk btool check --debug'
        Done
        Checking default conf files for edits...
        Validating installed files against hashes from '/opt/splunkforwarder/splunkforwarder-9.0.4-de405f4a7979-linux-2.6-x86_64-manifest'
        All installed files intact.
        Done
All preliminary checks passed.

Starting splunk server daemon (splunkd)...  
PYTHONHTTPSVERIFY is set to 0 in splunk-launch.conf disabling certificate validation for the httplib and urllib libraries shipped with the embedded Python interpreter; must be set to "1" for increased security
Done

[isoutamo]

Hi

I think that this alert_action.conf error is still under the fixing?

You could get ride of that execve error by disabling boot-start and then enabling it again?

r. Ismo

[blaha1]

I dont think Alpine includes systemd but uses OpenRC instead.

[spenna]

I have this same problem with containers. Works in 8.x, but get the same failure in 9.x. Investigating.

[spenna]

Adding the following to my compose file fixes the problem with docker containers in 9.x:

  splunk:
     tty: true

[edgars]

Thank you! This fixed the issue afret I upgraded from 8.x to 9.x.

[blaha1]

And if you are not using compose files, is there perhaps something that can be configured?

[spenna]

I don’t know if there is a config option for splunk itself. With docker cli, you should be able to add the -t flag and it would be the same as the compose version. 

[richgalloway]

I can't find it documented, but going from 7 to 9 may be too much of a jump.  Now that you're on 8, installing 9 should work.

[blaha1]

 Its not really an upgrade, I'm using docker containers so its basically a fresh install everytime so to speak.

[ShawnC]

That was rediculous to resctriction ternimal tty to start Splunkforwarder. If that how could the splunkforwarder process been contrulled under program manager like supervisor?

tty: true is a work aground way but not a good solution