splunk enterprise | services start error

[Mirza_Jaffar1]

why this issues I was trying to upgrade the splunk enterprise 

Checking prerequisites...
        Checking http port [8000]: open
        Checking mgmt port [8089]: open
        Checking appserver port [127.0.0.1:8065]: open
        Checking kvstore port [8191]: open
        Checking configuration... Done.
        Checking critical directories...        Done
        Checking indexes...
                Validated: _audit _configtracker _dsappevent _dsclient _dsphonehome _internal _introspection _metrics _metrics_rollup _telemetry _thefishbucket history main summary
        Done

Bypassing local license checks since this instance is configured with a remote license master.

        Checking filesystem compatibility...  Done
        Checking conf files for problems...
                Invalid key in stanza [email] in /opt/splunk/etc/apps/search/local/alert_actions.conf, line 2: show_password (value: True).
                Invalid key in stanza [cloud] in /opt/splunk/etc/apps/splunk_assist/default/assist.conf, line 14: http_client_timout_seconds (value: 30).
                Invalid key in stanza [setup] in /opt/splunk/etc/apps/splunk_secure_gateway/default/securegateway.conf, line 16: cluster_monitor_interval (value: 300).
                Invalid key in stanza [setup] in /opt/splunk/etc/apps/splunk_secure_gateway/default/securegateway.conf, line 20: cluster_mode_enabled (value: false).
                Your indexes and inputs configurations are not internally consistent. For more information, run 'splunk btool check --debug'
        Done
        Checking default conf files for edits...
        Validating installed files against hashes from '/opt/splunk/splunk-9.3.4-30e72d3fb5f7-linux-2.6-x86_64-manifest'
        All installed files intact.
        Done
All preliminary checks passed.

Starting splunk server daemon (splunkd)...
PYTHONHTTPSVERIFY is set to 0 in splunk-launch.conf disabling certificate validation for the httplib and urllib libraries shipped with the embedded Python interpreter; must be set to "1" for increased security
Done

Waiting for web server at https://127.0.0.1:8000 to be available.............splunkd 261927 was not running.
Stopping splunk he...

Done.
Stopped helpers.
Removing stale pid file... done.

WARNING: web interface does not seem to be available!

[Mirza_Jaffar1]

splunk and root permission conflicts as per the logs permission errors

1- wget version in /opt

2- .tgz allocate splunk permission

3- stop the splunk services

4- run tgz via splunk user while upgrdaing

This should work

[isoutamo]

Better to use RPM as then there are those pre and post scripts which are doing some cleaning etc. tasks which are not done if you are just unzipping that into /opt/splunk directory!

And  with tgz you must always do as root "chown -R splunk:splunk /opt/splunk" or whatever your splunk user is  before you start it after update!

[Mirza_Jaffar1]

this occurred While upgrading from the Splunk Enterprise v 8.2.8 ->9.1.0->9.2.0->9.3.0 to 9.3.4

[isoutamo]

Have you started your instance(s) every time after you have applied a new version? This is needed to make a needed conversions e.g. from 8.2.8 -> 9.1.0 etc.! Without those starts it’s almost same to do it directly 8.2.8 -> 9.3.4 especially if you are using tar.gz package. With rpm and deb installing a new, removing some old unneeded files too. But all conversion tasks have done only when you are starting the instance.

[livehybrid]

Hi @Mirza_Jaffar1 

What was the previous version and current version you are on now? Did you get a clean start when starting after upgrading to the previous version from the version before it?

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

[livehybrid]

Hi @Mirza_Jaffar1 

Something has failed in the startup process, please could you check your splunkd.log in $SPLUNK_HOME/var/log/splunk/splunkd.log and let us know what ERROR logs appear towards the end of the file?

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing