Installation

Upgrading Universal Forwarder 8.x.x to 9.x.x does not work?

blaha1
Explorer

I have been using the Universal  forwarder splunkforwarder-7.2.6-c0bf0f679ce9-Linux-x86_64 for quite a while without issues. I now wanted to upgrade to the latest one, 9.0.2 so I downloaded it and ran it just like I did with the old version. However, when starting it, 

${SPLUNK_HOME}/bin/splunk start --accept-license --answer-yes --no-prompt
 
It seems to crash with
 
Error calling execve(): No such file or directory
Error launching command: Invalid argument
 
I then tried the latest 8.x.x version, 8.2.9 and that worked perfectly fine.
 
What has changed between version 8 and 9? Any new requirements I am not aware of?
Labels (3)

blaha1
Explorer

I have now done some additional research and testing.

 

I am using Alpine Linux which does not include systemd. That is probably why this is not working for me.

 

 

8e23f2b85b3a:/# "/opt/splunkforwarder/bin/splunk" start --accept-license --answer-yes --no-prompt
Warning: Attempting to revert the SPLUNK_HOME ownership
Warning: Executing "chown -R splunk:splunk /opt/splunkforwarder"

This appears to be your first time running this version of Splunk.
Creating unit file...
Error calling execve(): No such file or directory
Error launching command: No such file or directory
Failed to create the unit file. Please do it manually later.


Splunk> The Notorious B.I.G. D.A.T.A.

Checking prerequisites...
Checking mgmt port [8089]: open
Creating: /opt/splunkforwarder/var/lib/splunk
Creating: /opt/splunkforwarder/var/run/splunk
Creating: /opt/splunkforwarder/var/run/splunk/appserver/i18n
Creating: /opt/splunkforwarder/var/run/splunk/appserver/modules/static/css
Creating: /opt/splunkforwarder/var/run/splunk/upload
Creating: /opt/splunkforwarder/var/run/splunk/search_telemetry
Creating: /opt/splunkforwarder/var/run/splunk/search_log
Creating: /opt/splunkforwarder/var/spool/splunk
Creating: /opt/splunkforwarder/var/spool/dirmoncache
Creating: /opt/splunkforwarder/var/lib/splunk/authDb
Creating: /opt/splunkforwarder/var/lib/splunk/hashDb
Checking conf files for problems...
Done
Checking default conf files for edits...
Validating installed files against hashes from '/opt/splunkforwarder/splunkforwarder-9.1.2-b6b9c8185839-linux-2.6-x86_64-manifest'
All installed files intact.
Done
All preliminary checks passed.

Starting splunk server daemon (splunkd)...
PYTHONHTTPSVERIFY is set to 0 in splunk-launch.conf disabling certificate validation for the httplib and urllib libraries shipped with the embedded Python interpreter; must be set to "1" for increased security

However it seems to start a background process but I dont see the logs in splunk. Using the status command kills the background process:

 

8e23f2b85b3a:/# "/opt/splunkforwarder/bin/splunk" status
Warning: Attempting to revert the SPLUNK_HOME ownership
Warning: Executing "chown -R splunk:splunk /opt/splunkforwarder"
splunkd 165 was not running.
Stopping splunk helpers...

 

I have tried disabling boot start:

splunk disable boot-start

But that gives me a similar error:

Error calling execve(): No such file or directory
Error launching command: No such file or directory
execve: No such file or directory
while running command /sbin/chkconfig

 

Has something changed from 8.x to 9.x that now systemd is used default somehow? How can I run the universal forwarder without systemd?

0 Karma

blaha1
Explorer

Still having this error with 9.0.4 I'm afraid.

 

 

50b81383ef0d:/opt/splunkforwarder/bin# ./splunk start --accept-license --answer-yes --no-prompt
Warning: Attempting to revert the SPLUNK_HOME ownership
Warning: Executing "chown -R splunk /opt/splunkforwarder"

This appears to be your first time running this version of Splunk.
Creating unit file...
Error calling execve(): No such file or directory
Error launching  command: No such file or directory
Failed to create the unit file. Please do it manually later.


Splunk> The Notorious B.I.G. D.A.T.A.

Checking prerequisites...
        Checking mgmt port [8089]: open
                Creating: /opt/splunkforwarder/var/lib/splunk
                Creating: /opt/splunkforwarder/var/run/splunk
                Creating: /opt/splunkforwarder/var/run/splunk/appserver/i18n
                Creating: /opt/splunkforwarder/var/run/splunk/appserver/modules/static/css
                Creating: /opt/splunkforwarder/var/run/splunk/upload
                Creating: /opt/splunkforwarder/var/run/splunk/search_telemetry
                Creating: /opt/splunkforwarder/var/spool/splunk
                Creating: /opt/splunkforwarder/var/spool/dirmoncache
                Creating: /opt/splunkforwarder/var/lib/splunk/authDb
                Creating: /opt/splunkforwarder/var/lib/splunk/hashDb
        Checking conf files for problems...
                Invalid key in stanza [webhook] in /opt/splunkforwarder/etc/system/default/alert_actions.conf, line 229: enable_allowlist (value: false).
                Your indexes and inputs configurations are not internally consistent. For more information, run 'splunk btool check --debug'
        Done
        Checking default conf files for edits...
        Validating installed files against hashes from '/opt/splunkforwarder/splunkforwarder-9.0.4-de405f4a7979-linux-2.6-x86_64-manifest'
        All installed files intact.
        Done
All preliminary checks passed.

Starting splunk server daemon (splunkd)...  
PYTHONHTTPSVERIFY is set to 0 in splunk-launch.conf disabling certificate validation for the httplib and urllib libraries shipped with the embedded Python interpreter; must be set to "1" for increased security
Done

 

 

 

0 Karma

isoutamo
SplunkTrust
SplunkTrust

Hi

I think that this alert_action.conf error is still under the fixing?

You could get ride of that execve error by disabling boot-start and then enabling it again?

r. Ismo

0 Karma

blaha1
Explorer

I dont think Alpine includes systemd but uses OpenRC instead.

Tags (1)
0 Karma

spenna
Explorer

I have this same problem with containers. Works in 8.x, but get the same failure in 9.x. Investigating.

0 Karma

spenna
Explorer

Adding the following to my compose file fixes the problem with docker containers in 9.x:

  splunk:
     tty: true

 

edgars
Explorer

Thank you! This fixed the issue afret I upgraded from 8.x to 9.x.

0 Karma

blaha1
Explorer

And if you are not using compose files, is there perhaps something that can be configured?

0 Karma

spenna
Explorer

I don’t know if there is a config option for splunk itself. With docker cli, you should be able to add the -t flag and it would be the same as the compose version. 

0 Karma

richgalloway
SplunkTrust
SplunkTrust

I can't find it documented, but going from 7 to 9 may be too much of a jump.  Now that you're on 8, installing 9 should work.

---
If this reply helps you, Karma would be appreciated.
0 Karma

blaha1
Explorer

 Its not really an upgrade, I'm using docker containers so its basically a fresh install everytime so to speak.

Tags (1)
0 Karma

ShawnC
Observer

That was rediculous to resctriction ternimal tty to start Splunkforwarder. If that how could the splunkforwarder process been contrulled under program manager like supervisor?

tty: true is a work aground way but not a good solution

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...