Installation

Upgrading Universal Forwarder 8.x.x to 9.x.x does not work?

blaha1
Explorer

I have been using the Universal  forwarder splunkforwarder-7.2.6-c0bf0f679ce9-Linux-x86_64 for quite a while without issues. I now wanted to upgrade to the latest one, 9.0.2 so I downloaded it and ran it just like I did with the old version. However, when starting it, 

${SPLUNK_HOME}/bin/splunk start --accept-license --answer-yes --no-prompt
 
It seems to crash with
 
Error calling execve(): No such file or directory
Error launching command: Invalid argument
 
I then tried the latest 8.x.x version, 8.2.9 and that worked perfectly fine.
 
What has changed between version 8 and 9? Any new requirements I am not aware of?
Labels (3)

blaha1
Explorer

I have now done some additional research and testing.

 

I am using Alpine Linux which does not include systemd. That is probably why this is not working for me.

 

 

8e23f2b85b3a:/# "/opt/splunkforwarder/bin/splunk" start --accept-license --answer-yes --no-prompt
Warning: Attempting to revert the SPLUNK_HOME ownership
Warning: Executing "chown -R splunk:splunk /opt/splunkforwarder"

This appears to be your first time running this version of Splunk.
Creating unit file...
Error calling execve(): No such file or directory
Error launching command: No such file or directory
Failed to create the unit file. Please do it manually later.


Splunk> The Notorious B.I.G. D.A.T.A.

Checking prerequisites...
Checking mgmt port [8089]: open
Creating: /opt/splunkforwarder/var/lib/splunk
Creating: /opt/splunkforwarder/var/run/splunk
Creating: /opt/splunkforwarder/var/run/splunk/appserver/i18n
Creating: /opt/splunkforwarder/var/run/splunk/appserver/modules/static/css
Creating: /opt/splunkforwarder/var/run/splunk/upload
Creating: /opt/splunkforwarder/var/run/splunk/search_telemetry
Creating: /opt/splunkforwarder/var/run/splunk/search_log
Creating: /opt/splunkforwarder/var/spool/splunk
Creating: /opt/splunkforwarder/var/spool/dirmoncache
Creating: /opt/splunkforwarder/var/lib/splunk/authDb
Creating: /opt/splunkforwarder/var/lib/splunk/hashDb
Checking conf files for problems...
Done
Checking default conf files for edits...
Validating installed files against hashes from '/opt/splunkforwarder/splunkforwarder-9.1.2-b6b9c8185839-linux-2.6-x86_64-manifest'
All installed files intact.
Done
All preliminary checks passed.

Starting splunk server daemon (splunkd)...
PYTHONHTTPSVERIFY is set to 0 in splunk-launch.conf disabling certificate validation for the httplib and urllib libraries shipped with the embedded Python interpreter; must be set to "1" for increased security

However it seems to start a background process but I dont see the logs in splunk. Using the status command kills the background process:

 

8e23f2b85b3a:/# "/opt/splunkforwarder/bin/splunk" status
Warning: Attempting to revert the SPLUNK_HOME ownership
Warning: Executing "chown -R splunk:splunk /opt/splunkforwarder"
splunkd 165 was not running.
Stopping splunk helpers...

 

I have tried disabling boot start:

splunk disable boot-start

But that gives me a similar error:

Error calling execve(): No such file or directory
Error launching command: No such file or directory
execve: No such file or directory
while running command /sbin/chkconfig

 

Has something changed from 8.x to 9.x that now systemd is used default somehow? How can I run the universal forwarder without systemd?

0 Karma

blaha1
Explorer

Still having this error with 9.0.4 I'm afraid.

 

 

50b81383ef0d:/opt/splunkforwarder/bin# ./splunk start --accept-license --answer-yes --no-prompt
Warning: Attempting to revert the SPLUNK_HOME ownership
Warning: Executing "chown -R splunk /opt/splunkforwarder"

This appears to be your first time running this version of Splunk.
Creating unit file...
Error calling execve(): No such file or directory
Error launching  command: No such file or directory
Failed to create the unit file. Please do it manually later.


Splunk> The Notorious B.I.G. D.A.T.A.

Checking prerequisites...
        Checking mgmt port [8089]: open
                Creating: /opt/splunkforwarder/var/lib/splunk
                Creating: /opt/splunkforwarder/var/run/splunk
                Creating: /opt/splunkforwarder/var/run/splunk/appserver/i18n
                Creating: /opt/splunkforwarder/var/run/splunk/appserver/modules/static/css
                Creating: /opt/splunkforwarder/var/run/splunk/upload
                Creating: /opt/splunkforwarder/var/run/splunk/search_telemetry
                Creating: /opt/splunkforwarder/var/spool/splunk
                Creating: /opt/splunkforwarder/var/spool/dirmoncache
                Creating: /opt/splunkforwarder/var/lib/splunk/authDb
                Creating: /opt/splunkforwarder/var/lib/splunk/hashDb
        Checking conf files for problems...
                Invalid key in stanza [webhook] in /opt/splunkforwarder/etc/system/default/alert_actions.conf, line 229: enable_allowlist (value: false).
                Your indexes and inputs configurations are not internally consistent. For more information, run 'splunk btool check --debug'
        Done
        Checking default conf files for edits...
        Validating installed files against hashes from '/opt/splunkforwarder/splunkforwarder-9.0.4-de405f4a7979-linux-2.6-x86_64-manifest'
        All installed files intact.
        Done
All preliminary checks passed.

Starting splunk server daemon (splunkd)...  
PYTHONHTTPSVERIFY is set to 0 in splunk-launch.conf disabling certificate validation for the httplib and urllib libraries shipped with the embedded Python interpreter; must be set to "1" for increased security
Done

 

 

 

0 Karma

isoutamo
SplunkTrust
SplunkTrust

Hi

I think that this alert_action.conf error is still under the fixing?

You could get ride of that execve error by disabling boot-start and then enabling it again?

r. Ismo

0 Karma

blaha1
Explorer

I dont think Alpine includes systemd but uses OpenRC instead.

Tags (1)
0 Karma

spenna
Explorer

I have this same problem with containers. Works in 8.x, but get the same failure in 9.x. Investigating.

0 Karma

spenna
Explorer

Adding the following to my compose file fixes the problem with docker containers in 9.x:

  splunk:
     tty: true

 

edgars
Explorer

Thank you! This fixed the issue afret I upgraded from 8.x to 9.x.

0 Karma

blaha1
Explorer

And if you are not using compose files, is there perhaps something that can be configured?

0 Karma

spenna
Explorer

I don’t know if there is a config option for splunk itself. With docker cli, you should be able to add the -t flag and it would be the same as the compose version. 

0 Karma

richgalloway
SplunkTrust
SplunkTrust

I can't find it documented, but going from 7 to 9 may be too much of a jump.  Now that you're on 8, installing 9 should work.

---
If this reply helps you, Karma would be appreciated.
0 Karma

blaha1
Explorer

 Its not really an upgrade, I'm using docker containers so its basically a fresh install everytime so to speak.

Tags (1)
0 Karma

ShawnC
Observer

That was rediculous to resctriction ternimal tty to start Splunkforwarder. If that how could the splunkforwarder process been contrulled under program manager like supervisor?

tty: true is a work aground way but not a good solution

0 Karma
Get Updates on the Splunk Community!

Take Your Breath Away with Splunk Risk-Based Alerting (RBA)

WATCH NOW!The Splunk Guide to Risk-Based Alerting is here to empower your SOC like never before. Join Haylee ...

SignalFlow: What? Why? How?

What is SignalFlow? Splunk Observability Cloud’s analytics engine, SignalFlow, opens up a world of in-depth ...

Federated Search for Amazon S3 | Key Use Cases to Streamline Compliance Workflows

Modern business operations are supported by data compliance. As regulations evolve, organizations must ...