topic Re: Upgrading Universal Forwarder 8.x.x to 9.x.x does not work? in Installation

Upgrading Universal Forwarder 8.x.x to 9.x.x does not work?

blaha1 — Mon, 07 Nov 2022 15:00:57 GMT

I have been using the Universal forwarder splunkforwarder-7.2.6-c0bf0f679ce9-Linux-x86_64 for quite a while without issues. I now wanted to upgrade to the latest one, 9.0.2 so I downloaded it and ran it just like I did with the old version. However, when starting it,

${SPLUNK_HOME}/bin/splunk start --accept-license --answer-yes --no-prompt

It seems to crash with

Error calling execve(): No such file or directory
Error launching command: Invalid argument

I then tried the latest 8.x.x version, 8.2.9 and that worked perfectly fine.

What has changed between version 8 and 9? Any new requirements I am not aware of?

Re: Upgrading Universal Forwarder 8.x.x to 9.x.x does not work?

richgalloway — Mon, 07 Nov 2022 15:39:25 GMT

I can't find it documented, but going from 7 to 9 may be too much of a jump. Now that you're on 8, installing 9 should work.

Re: Upgrading Universal Forwarder 8.x.x to 9.x.x does not work?

blaha1 — Mon, 07 Nov 2022 15:47:20 GMT

Its not really an upgrade, I'm using docker containers so its basically a fresh install everytime so to speak.

Re: Upgrading Universal Forwarder 8.x.x to 9.x.x does not work?

spenna — Thu, 24 Nov 2022 01:11:19 GMT

I have this same problem with containers. Works in 8.x, but get the same failure in 9.x. Investigating.

Re: Upgrading Universal Forwarder 8.x.x to 9.x.x does not work?

spenna — Thu, 24 Nov 2022 01:20:53 GMT

Adding the following to my compose file fixes the problem with docker containers in 9.x:

splunk: tty: true

Re: Upgrading Universal Forwarder 8.x.x to 9.x.x does not work?

blaha1 — Thu, 24 Nov 2022 07:26:42 GMT

And if you are not using compose files, is there perhaps something that can be configured?

Re: Upgrading Universal Forwarder 8.x.x to 9.x.x does not work?

spenna — Thu, 24 Nov 2022 12:11:35 GMT

I don’t know if there is a config option for splunk itself. With docker cli, you should be able to add the -t flag and it would be the same as the compose version.

Re: Upgrading Universal Forwarder 8.x.x to 9.x.x does not work?

edgars — Wed, 08 Feb 2023 16:00:18 GMT

Thank you! This fixed the issue afret I upgraded from 8.x to 9.x.

Re: Upgrading Universal Forwarder 8.x.x to 9.x.x does not work?

blaha1 — Wed, 14 Feb 2024 14:43:47 GMT

Still having this error with 9.0.4 I'm afraid.

50b81383ef0d:/opt/splunkforwarder/bin# ./splunk start --accept-license --answer-yes --no-prompt Warning: Attempting to revert the SPLUNK_HOME ownership Warning: Executing "chown -R splunk /opt/splunkforwarder" This appears to be your first time running this version of Splunk. Creating unit file... Error calling execve(): No such file or directory Error launching command: No such file or directory Failed to create the unit file. Please do it manually later. Splunk> The Notorious B.I.G. D.A.T.A. Checking prerequisites... Checking mgmt port [8089]: open Creating: /opt/splunkforwarder/var/lib/splunk Creating: /opt/splunkforwarder/var/run/splunk Creating: /opt/splunkforwarder/var/run/splunk/appserver/i18n Creating: /opt/splunkforwarder/var/run/splunk/appserver/modules/static/css Creating: /opt/splunkforwarder/var/run/splunk/upload Creating: /opt/splunkforwarder/var/run/splunk/search_telemetry Creating: /opt/splunkforwarder/var/spool/splunk Creating: /opt/splunkforwarder/var/spool/dirmoncache Creating: /opt/splunkforwarder/var/lib/splunk/authDb Creating: /opt/splunkforwarder/var/lib/splunk/hashDb Checking conf files for problems... Invalid key in stanza [webhook] in /opt/splunkforwarder/etc/system/default/alert_actions.conf, line 229: enable_allowlist (value: false). Your indexes and inputs configurations are not internally consistent. For more information, run 'splunk btool check --debug' Done Checking default conf files for edits... Validating installed files against hashes from '/opt/splunkforwarder/splunkforwarder-9.0.4-de405f4a7979-linux-2.6-x86_64-manifest' All installed files intact. Done All preliminary checks passed. Starting splunk server daemon (splunkd)... PYTHONHTTPSVERIFY is set to 0 in splunk-launch.conf disabling certificate validation for the httplib and urllib libraries shipped with the embedded Python interpreter; must be set to "1" for increased security Done

Re: Upgrading Universal Forwarder 8.x.x to 9.x.x does not work?

isoutamo — Fri, 13 Oct 2023 10:50:13 GMT

I think that this alert_action.conf error is still under the fixing?

You could get ride of that execve error by disabling boot-start and then enabling it again?

r. Ismo

Re: Upgrading Universal Forwarder 8.x.x to 9.x.x does not work?

blaha1 — Fri, 20 Oct 2023 11:31:37 GMT

I dont think Alpine includes systemd but uses OpenRC instead.

Re: Upgrading Universal Forwarder 8.x.x to 9.x.x does not work?

blaha1 — Thu, 15 Feb 2024 10:59:14 GMT

I have now done some additional research and testing.

I am using Alpine Linux which does not include systemd. That is probably why this is not working for me.

8e23f2b85b3a:/# "/opt/splunkforwarder/bin/splunk" start --accept-license --answer-yes --no-prompt
Warning: Attempting to revert the SPLUNK_HOME ownership
Warning: Executing "chown -R splunk:splunk /opt/splunkforwarder"

This appears to be your first time running this version of Splunk.
Creating unit file...
Error calling execve(): No such file or directory
Error launching command: No such file or directory
Failed to create the unit file. Please do it manually later.

Splunk> The Notorious B.I.G. D.A.T.A.

Checking prerequisites...
Checking mgmt port [8089]: open
Creating: /opt/splunkforwarder/var/lib/splunk
Creating: /opt/splunkforwarder/var/run/splunk
Creating: /opt/splunkforwarder/var/run/splunk/appserver/i18n
Creating: /opt/splunkforwarder/var/run/splunk/appserver/modules/static/css
Creating: /opt/splunkforwarder/var/run/splunk/upload
Creating: /opt/splunkforwarder/var/run/splunk/search_telemetry
Creating: /opt/splunkforwarder/var/run/splunk/search_log
Creating: /opt/splunkforwarder/var/spool/splunk
Creating: /opt/splunkforwarder/var/spool/dirmoncache
Creating: /opt/splunkforwarder/var/lib/splunk/authDb
Creating: /opt/splunkforwarder/var/lib/splunk/hashDb
Checking conf files for problems...
Done
Checking default conf files for edits...
Validating installed files against hashes from '/opt/splunkforwarder/splunkforwarder-9.1.2-b6b9c8185839-linux-2.6-x86_64-manifest'
All installed files intact.
Done
All preliminary checks passed.

Starting splunk server daemon (splunkd)...
PYTHONHTTPSVERIFY is set to 0 in splunk-launch.conf disabling certificate validation for the httplib and urllib libraries shipped with the embedded Python interpreter; must be set to "1" for increased security

However it seems to start a background process but I dont see the logs in splunk. Using the status command kills the background process:

8e23f2b85b3a:/# "/opt/splunkforwarder/bin/splunk" status
Warning: Attempting to revert the SPLUNK_HOME ownership
Warning: Executing "chown -R splunk:splunk /opt/splunkforwarder"
splunkd 165 was not running.
Stopping splunk helpers...

I have tried disabling boot start:

splunk disable boot-start

But that gives me a similar error:

Error calling execve(): No such file or directory
Error launching command: No such file or directory
execve: No such file or directory
while running command /sbin/chkconfig

Has something changed from 8.x to 9.x that now systemd is used default somehow? How can I run the universal forwarder without systemd?

Re: Upgrading Universal Forwarder 8.x.x to 9.x.x does not work?

ShawnC — Wed, 27 Mar 2024 03:57:23 GMT

That was rediculous to resctriction ternimal tty to start Splunkforwarder. If that how could the splunkforwarder process been contrulled under program manager like supervisor?

tty: true is a work aground way but not a good solution