500 Internal server error

[alvaroveiga]

After upgrading to latest Splunk enterprise version, i'am getting this error:
https://image.ibb.co/mbpbuQ/1.jpg

btool check --debug:
No spec file for: /opt/splunk/etc/apps/FileServ/default/fileserv.conf
Improper stanza [dhcpd_server_dhcprelease] in /opt/splunk/etc/apps/unix/default/tags.conf, line 30
Invalid key in stanza [email] in /opt/splunk/etc/system/local/alert_actions.conf, line 5: reportServerEnabled (value: 1).
Did you mean 'reportCIDFontList'?
Did you mean 'reportFileName'?
Did you mean 'reportIncludeSplunkLogo'?
Did you mean 'reportPaperOrientation'?
Did you mean 'reportPaperSize'?
Invalid key in stanza [email] in /opt/splunk/etc/system/local/alert_actions.conf, line 6: reportServerURL (value: ).
Did you mean 'reportCIDFontList'?
Did you mean 'reportFileName'?
Did you mean 'reportIncludeSplunkLogo'?
Did you mean 'reportPaperOrientation'?
Did you mean 'reportPaperSize'?
Checking: /opt/splunk/etc/system/local/authentication.conf
Checking: /opt/splunk/etc/system/local/authorize.conf
Checking: /opt/splunk/etc/system/local/distsearch.conf
Checking: /opt/splunk/etc/system/local/eventtypes.conf
Checking: /opt/splunk/etc/system/local/indexes.conf
Checking: /opt/splunk/etc/system/local/inputs.conf
No spec file for: /opt/splunk/etc/system/local/migration.conf
Checking: /opt/splunk/etc/system/local/props.conf
Checking: /opt/splunk/etc/system/local/server.conf
Checking: /opt/splunk/etc/system/local/serverclass.conf
No spec file for: /opt/splunk/etc/system/local/tenants.conf
Checking: /opt/splunk/etc/system/local/transforms.conf
Checking: /opt/splunk/etc/system/local/web.conf

How can i fix that?

[lfedak_splunk]

Hey @alvaroveiga, if @mwdbhyat or @harsmarvania57 answered your question please remember to accept their answer. You can upvote posts as well. (Karma points will be awarded for either action.) Happy Splunking!

[harsmarvania57]

While looking at error it looks like reportServerEnabled and reportServerURL parameter in your alert_actions.conf does not support in Splunk 6.6.3. Please refer http://docs.splunk.com/Documentation/Splunk/6.6.3/Admin/alertactionsconf

So please remove those 2 parameters from alert_actions.conf configuration file and try to start splunk again.

Thanks,
Harshil

[alvaroveiga]

Still same problem.

bin]# ./splunk start

Splunk> The Notorious B.I.G. D.A.T.A.

Checking prerequisites...
Checking http port [10.244.161.7:8000]: open
Checking mgmt port [10.244.161.7:8089]: open
Checking appserver port [127.0.0.1:8065]: open
Checking kvstore port [10.244.161.7:8191]: open
Checking configuration... Done.
Checking critical directories... Done
Checking indexes...
Validated: _audit _internal _introspection _telemetry _thefishbu cket checkfwd eqalis_network_sample firewall history itau main mwg_audit os osse c perfmon snort_cardholder snort_servidores sos sos_summary_daily summary summar y_forwarders summary_hosts summary_indexers summary_pools summary_sources summar y_sourcetypes syslog tp_win_sec tp_win_servers windows wineventlog
Done

Bypassing local license checks since this instance is configured with a remote l icense master.

    Checking filesystem compatibility...  Done
    Checking conf files for problems...
            Improper stanza [dhcpd_server_dhcprelease] in /opt/splunk/etc/ap                                                                                                                                                                                                                                             ps/unix/default/tags.conf, line 30
            Your indexes and inputs configurations are not internally consis                                                                                                                                                                                                                                             tent. For more information, run 'splunk btool check --debug'
    Done
    Checking default conf files for edits...
    Validating installed files against hashes from '/opt/splunk/splunk-6.6.3                                                                                                                                                                                                                                             -e21ee54bc796-linux-2.6-x86_64-manifest'
    All installed files intact.
    Done

All preliminary checks passed.

Starting splunk server daemon (splunkd)...
Done
[ OK ]

Waiting for web server at https://10.244.161.7:8000 to be available... Done

If you get stuck, we're here to help.
Look for answers here: http://docs.splunk.com

The Splunk web interface is at https://10.244.161.7:8000

[harsmarvania57]

Splunk started perfectly fine, only one warning message because you are using old version of Splunk App for Unix and Linux. Please upgrade that, you might need to remove old unix app because new app folder name has been chanegd to splunk_app_for_nix

[alvaroveiga]

After i log with my credentials i get an "500 Internal Error" doesnt matter the username.

[harsmarvania57]

Any error in $SPLUNK_HOME/var/log/splunk/web_service.log ?

[alvaroveiga]

Yes
link text

[alvaroveiga]

Yes.

Pastebin log

[alvaroveiga]

https://pastebin.com/iBN3aLV2

[harsmarvania57]

Looks like some cherrypy session related problem , I'll suggest to open case with splunk support.

[mwdbhyat]

Is https still being used after the upgrade? Try http - it could have reset or not loading conf files properly. Or is this message only occurring when trying to load a certain page?

[alvaroveiga]

http doesnt work, only https.
The error occur after i login with my credentials.
I dont know what to do.

*> # ./splunk start

Splunk> Now with more code!

Checking prerequisites...
Checking http port [10.244.161.7:8000]: open
Checking mgmt port [10.244.161.7:8089]: open
Checking appserver port [127.0.0.1:8065]: open
Checking kvstore port [10.244.161.7:8191]: open
Checking configuration... Done.
Checking critical directories... Done
Checking indexes...
Validated: _audit _internal _introspection _telemetry _thefishbu cket checkfwd eqalis_network_sample
firewall history itau main mwg_audit
os osse

c perfmon snort_cardholder
snort_servidores sos sos_summary_daily
summary summar

y_forwarders summary_hosts
summary_indexers summary_pools
summary_sources summar

y_sourcetypes syslog tp_win_sec
tp_win_servers windows wineventlog
Done

Bypassing local license checks since
this instance is configured with a
remote l

icense master.

    Checking filesystem compatibility...  Done
    Checking conf files for problems...
            Improper stanza [dhcpd_server_dhcprelease] in

/opt/splunk/etc/ap

ps/unix/default/tags.conf, line 30
Invalid key in stanza [email] in
/opt/splunk/etc/system/local/al

ert_actions.conf, line 5:
reportServerEnabled (value: 1).
Invalid key in stanza [email] in
/opt/splunk/etc/system/local/al

ert_actions.conf, line 6:
reportServerURL (value: ).
Your indexes and inputs configurations are not
internally consis

tent. For more information, run
'splunk btool check --debug'
Done
Checking default conf files for edits...
Validating installed files against hashes from
'/opt/splunk/splunk-6.6.3

-e21ee54bc796-linux-2.6-x86_64-manifest'
All installed files intact.
Done All preliminary checks passed.

Starting splunk server daemon
(splunkd)... Done
[ OK ]

Waiting for web server at
https://10.244.161.7:8000 to be
available... Done

If you get stuck, we're here to help.
Look for answers here:
http://docs.splunk.com

The Splunk web interface is at
https://10.244.161.7:8000*

[mwdbhyat]

Are other users experiencing the same problem or just you? It could be ssl related. Are you running in a distributed environment, can you log in to other servers?

I can see you have a few conf errors as well - this link should help you fix those old values for the conf files:
https://answers.splunk.com/answers/548915/after-a-successful-upgrade-from-621-to-661-we-are.html

[alvaroveiga]

All users on the same splunk indexer are experiencing the error, the others with older enterprise version have no problem.
What can be causing it? The server only runs splunk enterprise.