topic Re: 500 Internal server error in Deployment Architecture

500 Internal server error

alvaroveiga — Tue, 29 Sep 2020 15:55:44 GMT

After upgrading to latest Splunk enterprise version, i'am getting this error:
https://image.ibb.co/mbpbuQ/1.jpg

btool check --debug:
No spec file for: /opt/splunk/etc/apps/FileServ/default/fileserv.conf
Improper stanza [dhcpd_server_dhcprelease] in /opt/splunk/etc/apps/unix/default/tags.conf, line 30
Invalid key in stanza [email] in /opt/splunk/etc/system/local/alert_actions.conf, line 5: reportServerEnabled (value: 1).
Did you mean 'reportCIDFontList'?
Did you mean 'reportFileName'?
Did you mean 'reportIncludeSplunkLogo'?
Did you mean 'reportPaperOrientation'?
Did you mean 'reportPaperSize'?
Invalid key in stanza [email] in /opt/splunk/etc/system/local/alert_actions.conf, line 6: reportServerURL (value: ).
Did you mean 'reportCIDFontList'?
Did you mean 'reportFileName'?
Did you mean 'reportIncludeSplunkLogo'?
Did you mean 'reportPaperOrientation'?
Did you mean 'reportPaperSize'?
Checking: /opt/splunk/etc/system/local/authentication.conf
Checking: /opt/splunk/etc/system/local/authorize.conf
Checking: /opt/splunk/etc/system/local/distsearch.conf
Checking: /opt/splunk/etc/system/local/eventtypes.conf
Checking: /opt/splunk/etc/system/local/indexes.conf
Checking: /opt/splunk/etc/system/local/inputs.conf
No spec file for: /opt/splunk/etc/system/local/migration.conf
Checking: /opt/splunk/etc/system/local/props.conf
Checking: /opt/splunk/etc/system/local/server.conf
Checking: /opt/splunk/etc/system/local/serverclass.conf
No spec file for: /opt/splunk/etc/system/local/tenants.conf
Checking: /opt/splunk/etc/system/local/transforms.conf
Checking: /opt/splunk/etc/system/local/web.conf

How can i fix that?

Re: 500 Internal server error

mwdbhyat — Mon, 25 Sep 2017 15:43:19 GMT

Is https still being used after the upgrade? Try http - it could have reset or not loading conf files properly. Or is this message only occurring when trying to load a certain page?

Re: 500 Internal server error

alvaroveiga — Tue, 29 Sep 2020 15:56:11 GMT

http doesnt work, only https.
The error occur after i login with my credentials.
I dont know what to do.

*> # ./splunk start

Splunk> Now with more code!

Checking prerequisites...
Checking http port [10.244.161.7:8000]: open
Checking mgmt port [10.244.161.7:8089]: open
Checking appserver port [127.0.0.1:8065]: open
Checking kvstore port [10.244.161.7:8191]: open
Checking configuration... Done.
Checking critical directories... Done
Checking indexes...
Validated: _audit _internal _introspection _telemetry _thefishbu cket checkfwd eqalis_network_sample
firewall history itau main mwg_audit
os osse

c perfmon snort_cardholder
snort_servidores sos sos_summary_daily
summary summar

y_forwarders summary_hosts
summary_indexers summary_pools
summary_sources summar

y_sourcetypes syslog tp_win_sec
tp_win_servers windows wineventlog
Done

Bypassing local license checks since
this instance is configured with a
remote l

icense master.
    Checking filesystem compatibility...  Done
    Checking conf files for problems...
            Improper stanza [dhcpd_server_dhcprelease] in
/opt/splunk/etc/ap

ps/unix/default/tags.conf, line 30
Invalid key in stanza [email] in
/opt/splunk/etc/system/local/al

ert_actions.conf, line 5:
reportServerEnabled (value: 1).
Invalid key in stanza [email] in
/opt/splunk/etc/system/local/al

ert_actions.conf, line 6:
reportServerURL (value: ).
Your indexes and inputs configurations are not
internally consis

tent. For more information, run
'splunk btool check --debug'
Done
Checking default conf files for edits...
Validating installed files against hashes from
'/opt/splunk/splunk-6.6.3

-e21ee54bc796-linux-2.6-x86_64-manifest'
All installed files intact.
Done All preliminary checks passed.

Starting splunk server daemon
(splunkd)... Done
[ OK ]

Waiting for web server at
https://10.244.161.7:8000 to be
available... Done

If you get stuck, we're here to help.
Look for answers here:
http://docs.splunk.com

The Splunk web interface is at
https://10.244.161.7:8000*

Re: 500 Internal server error

mwdbhyat — Mon, 25 Sep 2017 18:26:00 GMT

Are other users experiencing the same problem or just you? It could be ssl related. Are you running in a distributed environment, can you log in to other servers?

I can see you have a few conf errors as well - this link should help you fix those old values for the conf files:
https://answers.splunk.com/answers/548915/after-a-successful-upgrade-from-621-to-661-we-are.html

Re: 500 Internal server error

alvaroveiga — Mon, 25 Sep 2017 18:46:01 GMT

All users on the same splunk indexer are experiencing the error, the others with older enterprise version have no problem.
What can be causing it? The server only runs splunk enterprise.

Re: 500 Internal server error

harsmarvania57 — Tue, 26 Sep 2017 15:55:13 GMT

While looking at error it looks like reportServerEnabled and reportServerURL parameter in your alert_actions.conf does not support in Splunk 6.6.3. Please refer http://docs.splunk.com/Documentation/Splunk/6.6.3/Admin/alertactionsconf

So please remove those 2 parameters from alert_actions.conf configuration file and try to start splunk again.

Thanks,
Harshil

Re: 500 Internal server error

lfedak_splunk — Tue, 26 Sep 2017 21:46:32 GMT

Hey @alvaroveiga, if @mwdbhyat or @harsmarvania57 answered your question please remember to accept their answer. You can upvote posts as well. (Karma points will be awarded for either action.) Happy Splunking!

Re: 500 Internal server error

alvaroveiga — Tue, 29 Sep 2020 16:02:18 GMT

Still same problem.

bin]# ./splunk start

Splunk> The Notorious B.I.G. D.A.T.A.

Checking prerequisites...
Checking http port [10.244.161.7:8000]: open
Checking mgmt port [10.244.161.7:8089]: open
Checking appserver port [127.0.0.1:8065]: open
Checking kvstore port [10.244.161.7:8191]: open
Checking configuration... Done.
Checking critical directories... Done
Checking indexes...
Validated: _audit _internal _introspection _telemetry _thefishbu cket checkfwd eqalis_network_sample firewall history itau main mwg_audit os osse c perfmon snort_cardholder snort_servidores sos sos_summary_daily summary summar y_forwarders summary_hosts summary_indexers summary_pools summary_sources summar y_sourcetypes syslog tp_win_sec tp_win_servers windows wineventlog
Done

Bypassing local license checks since this instance is configured with a remote l icense master.

    Checking filesystem compatibility...  Done
    Checking conf files for problems...
            Improper stanza [dhcpd_server_dhcprelease] in /opt/splunk/etc/ap                                                                                                                                                                                                                                             ps/unix/default/tags.conf, line 30
            Your indexes and inputs configurations are not internally consis                                                                                                                                                                                                                                             tent. For more information, run 'splunk btool check --debug'
    Done
    Checking default conf files for edits...
    Validating installed files against hashes from '/opt/splunk/splunk-6.6.3                                                                                                                                                                                                                                             -e21ee54bc796-linux-2.6-x86_64-manifest'
    All installed files intact.
    Done

All preliminary checks passed.

Starting splunk server daemon (splunkd)...
Done
[ OK ]

Waiting for web server at https://10.244.161.7:8000 to be available... Done

If you get stuck, we're here to help.
Look for answers here: http://docs.splunk.com

The Splunk web interface is at https://10.244.161.7:8000

Re: 500 Internal server error

harsmarvania57 — Tue, 29 Sep 2020 15:58:06 GMT

Splunk started perfectly fine, only one warning message because you are using old version of Splunk App for Unix and Linux. Please upgrade that, you might need to remove old unix app because new app folder name has been chanegd to splunk_app_for_nix

Re: 500 Internal server error

alvaroveiga — Wed, 27 Sep 2017 14:50:24 GMT

After i log with my credentials i get an "500 Internal Error" doesnt matter the username.

Re: 500 Internal server error

harsmarvania57 — Tue, 29 Sep 2020 15:58:12 GMT

Any error in $SPLUNK_HOME/var/log/splunk/web_service.log ?

Re: 500 Internal server error

alvaroveiga — Wed, 27 Sep 2017 17:23:16 GMT

Yes
link text

Re: 500 Internal server error

alvaroveiga — Wed, 27 Sep 2017 17:23:38 GMT

Yes.

Pastebin log

Re: 500 Internal server error

alvaroveiga — Wed, 27 Sep 2017 17:23:46 GMT

https://pastebin.com/iBN3aLV2

Re: 500 Internal server error

harsmarvania57 — Thu, 28 Sep 2017 11:43:31 GMT

Looks like some cherrypy session related problem , I'll suggest to open case with splunk support.