Upgrading Universal Forwarder 8.x.x to 9.x.x does not work?

[blaha1]

I have been using the Universal  forwarder splunkforwarder-7.2.6-c0bf0f679ce9-Linux-x86_64 for quite a while without issues. I now wanted to upgrade to the latest one, 9.0.2 so I downloaded it and ran it just like I did with the old version. However, when starting it, 

${SPLUNK_HOME}/bin/splunk start --accept-license --answer-yes --no-prompt

It seems to crash with

Error calling execve(): No such file or directory
Error launching command: Invalid argument

I then tried the latest 8.x.x version, 8.2.9 and that worked perfectly fine.

What has changed between version 8 and 9? Any new requirements I am not aware of?

[blaha1]

Still having this error with 9.0.4 I'm afraid.

50b81383ef0d:/opt/splunkforwarder/bin# ./splunk start --accept-license --answer-yes --no-prompt
Warning: Attempting to revert the SPLUNK_HOME ownership
Warning: Executing "chown -R splunk /opt/splunkforwarder"

This appears to be your first time running this version of Splunk.
Creating unit file...
Error calling execve(): No such file or directory
Error launching  command: No such file or directory
Failed to create the unit file. Please do it manually later.


Splunk> The Notorious B.I.G. D.A.T.A.

Checking prerequisites...
        Checking mgmt port [8089]: open
                Creating: /opt/splunkforwarder/var/lib/splunk
                Creating: /opt/splunkforwarder/var/run/splunk
                Creating: /opt/splunkforwarder/var/run/splunk/appserver/i18n
                Creating: /opt/splunkforwarder/var/run/splunk/appserver/modules/static/css
                Creating: /opt/splunkforwarder/var/run/splunk/upload
                Creating: /opt/splunkforwarder/var/run/splunk/search_telemetry
                Creating: /opt/splunkforwarder/var/spool/splunk
                Creating: /opt/splunkforwarder/var/spool/dirmoncache
                Creating: /opt/splunkforwarder/var/lib/splunk/authDb
                Creating: /opt/splunkforwarder/var/lib/splunk/hashDb
        Checking conf files for problems...
                Invalid key in stanza [webhook] in /opt/splunkforwarder/etc/system/default/alert_actions.conf, line 229: enable_allowlist (value: false).
                Your indexes and inputs configurations are not internally consistent. For more information, run 'splunk btool check --debug'
        Done
        Checking default conf files for edits...
        Validating installed files against hashes from '/opt/splunkforwarder/splunkforwarder-9.0.4-de405f4a7979-linux-2.6-x86_64-manifest'
        All installed files intact.
        Done
All preliminary checks passed.

Starting splunk server daemon (splunkd)...  
PYTHONHTTPSVERIFY is set to 0 in splunk-launch.conf disabling certificate validation for the httplib and urllib libraries shipped with the embedded Python interpreter; must be set to "1" for increased security
Done

However, it seems to at least startup now and I can see in my splunk dashboard that logs are indeed coming in. So it does work but I have these errors.

[isoutamo]

Hi

I think that this alert_action.conf error is still under the fixing?

You could get ride of that execve error by disabling boot-start and then enabling it again?

r. Ismo

[blaha1]

I dont think Alpine includes systemd but uses OpenRC instead.

[blaha1]

If I run disable boot-start before I start it, this happens:

"${SPLUNK_HOME}/bin/splunk" disable boot-start
Error calling execve(): No such file or directory
Error launching  command: No such file or directory
execve: No such file or directory
  while running command /sbin/chkconfig

[blaha1]

Hmm can you explain a bit more? What is this alert_action.conf doing? Not sure what you mean about the boot start thing. As I am running a docker container, it always boots and runs the startup command.   How do I disable it?

[spenna]

I have this same problem with containers. Works in 8.x, but get the same failure in 9.x. Investigating.

[spenna]

Adding the following to my compose file fixes the problem with docker containers in 9.x:

  splunk:
     tty: true

[edgars]

Thank you! This fixed the issue afret I upgraded from 8.x to 9.x.

[blaha1]

And if you are not using compose files, is there perhaps something that can be configured?

[spenna]

I don’t know if there is a config option for splunk itself. With docker cli, you should be able to add the -t flag and it would be the same as the compose version. 

[richgalloway]

I can't find it documented, but going from 7 to 9 may be too much of a jump.  Now that you're on 8, installing 9 should work.

[blaha1]

 Its not really an upgrade, I'm using docker containers so its basically a fresh install everytime so to speak.