Splunk not starting after upgrade (6.6.1 > 7.0.0).

[alvaroveiga]

Hi, i just updated from 6.6.1 to latest version(7) and now i'am stuck with splunk not starting web interface:

./splunk restart

Stopping splunkd...
Shutting down. Please wait, as this may take a few minutes.
..................................... [ OK ]
Stopping splunk helpers...
[ OK ]
Done.

Splunk> Map. Reduce. Recycle.

Checking prerequisites...
Checking http port [10.244.161.7:8000]: open
Checking mgmt port [10.244.161.7:8089]: open
Checking appserver port [127.0.0.1:8065]: open
Checking kvstore port [10.244.161.7:8191]: open
Checking configuration... Done.
Checking critical directories... Done
Checking indexes...
Validated: _audit _internal _introspection _telemetry _thefishbucket checkfwd eqalis_network_sample firewall history itau main mwg_audit os ossec perfmon snort_cardholder snort_servidores sos sos_summary_daily summary summary_forwarders summary_hosts summary_indexers summary_pools summary_sources summary_sourcetypes syslog tp_win_sec tp_win_servers windows wineventlog
Done

Bypassing local license checks since this instance is configured with a remote license master.

    Checking filesystem compatibility...  Done
    Checking conf files for problems...
            Improper stanza [dhcpd_server_dhcprelease] in /opt/splunk/etc/apps/unix/default/tags.conf, line 30
            Your indexes and inputs configurations are not internally consistent. For more information, run 'splunk btool check --debug'
    Done
    Checking default conf files for edits...
    Validating installed files against hashes from '/opt/splunk/splunk-7.0.0-c8a78efdd40f-linux-2.6-x86_64-manifest'
    All installed files intact.
    Done

All preliminary checks passed.

Starting splunk server daemon (splunkd)...
Done
[ OK ]

Waiting for web server at https://10.244.161.7:8000 to be available............................................................................................................................................................................................................................................................................................................

WARNING: web interface does not seem to be available!

What can be causing it ?

[bgagliardi1]

I had the same issue and I had to look at the crash log and found (in hex code) that there was a duplicate HEC (HTTP Event Collector) key in an app. So in summary, I had an app that was a culprit. You can backup all your apps, and either remove all and add 1 at a time and restart splunk, or have them all on there and delete one by 1 and try starting splunk.

This is the process I went through and it is also the recommended approach by Splunk to ensure that all apps work on a splunk (dev) server before upgrading prod.

[bgagliardi1]

I was coming from 6.6.2 to 7.0.3

There were no logs in splunkd or the web logs.

[everagu]

I have the same message after upgrade, just wait a minute and try start it again. That works for me.

[alvaroveiga]

Didnt work.

[Sukisen1981]

Tried - https://10.244.161.7:8001?

[alvaroveiga]

Doesnt work..

[harsmarvania57]

Have you tried to access https://10.244.161.7:8000 ? If it's not working then any error logs in $SPLUNK_HOME/var/log/splunk/web_service.log ?

[alvaroveiga]

Not working, here is the logs:
https://pastebin.com/3Z5pmzCs

Could you please help me understanding it?