Deployment Architecture

500 Internal server error

New Member

After upgrading to latest Splunk enterprise version, i'am getting this error:
https://image.ibb.co/mbpbuQ/1.jpg

btool check --debug:
No spec file for: /opt/splunk/etc/apps/FileServ/default/fileserv.conf
Improper stanza [dhcpdserverdhcprelease] in /opt/splunk/etc/apps/unix/default/tags.conf, line 30
Invalid key in stanza [email] in /opt/splunk/etc/system/local/alertactions.conf, line 5: reportServerEnabled (value: 1).
Did you mean 'reportCIDFontList'?
Did you mean 'reportFileName'?
Did you mean 'reportIncludeSplunkLogo'?
Did you mean 'reportPaperOrientation'?
Did you mean 'reportPaperSize'?
Invalid key in stanza [email] in /opt/splunk/etc/system/local/alert
actions.conf, line 6: reportServerURL (value: ).
Did you mean 'reportCIDFontList'?
Did you mean 'reportFileName'?
Did you mean 'reportIncludeSplunkLogo'?
Did you mean 'reportPaperOrientation'?
Did you mean 'reportPaperSize'?
Checking: /opt/splunk/etc/system/local/authentication.conf
Checking: /opt/splunk/etc/system/local/authorize.conf
Checking: /opt/splunk/etc/system/local/distsearch.conf
Checking: /opt/splunk/etc/system/local/eventtypes.conf
Checking: /opt/splunk/etc/system/local/indexes.conf
Checking: /opt/splunk/etc/system/local/inputs.conf
No spec file for: /opt/splunk/etc/system/local/migration.conf
Checking: /opt/splunk/etc/system/local/props.conf
Checking: /opt/splunk/etc/system/local/server.conf
Checking: /opt/splunk/etc/system/local/serverclass.conf
No spec file for: /opt/splunk/etc/system/local/tenants.conf
Checking: /opt/splunk/etc/system/local/transforms.conf
Checking: /opt/splunk/etc/system/local/web.conf

How can i fix that?

0 Karma

Splunk Employee
Splunk Employee

Hey @alvaroveiga, if @mwdbhyat or @harsmarvania57 answered your question please remember to accept their answer. You can upvote posts as well. (Karma points will be awarded for either action.) Happy Splunking!

0 Karma

SplunkTrust
SplunkTrust

While looking at error it looks like reportServerEnabled and reportServerURL parameter in your alert_actions.conf does not support in Splunk 6.6.3. Please refer http://docs.splunk.com/Documentation/Splunk/6.6.3/Admin/alertactionsconf

So please remove those 2 parameters from alert_actions.conf configuration file and try to start splunk again.

Thanks,
Harshil

0 Karma

New Member

Still same problem.

bin]# ./splunk start

Splunk> The Notorious B.I.G. D.A.T.A.

Checking prerequisites...
Checking http port [10.244.161.7:8000]: open
Checking mgmt port [10.244.161.7:8089]: open
Checking appserver port [127.0.0.1:8065]: open
Checking kvstore port [10.244.161.7:8191]: open
Checking configuration... Done.
Checking critical directories... Done
Checking indexes...
Validated: audit _internal _introspection _telemetry _thefishbu cket checkfwd eqalisnetworksample firewall history itau main mwgaudit os osse c perfmon snortcardholder snortservidores sos sossummarydaily summary summar yforwarders summaryhosts summaryindexers summarypools summarysources summar ysourcetypes syslog tpwinsec tpwinservers windows wineventlog
Done

Bypassing local license checks since this instance is configured with a remote l icense master.

    Checking filesystem compatibility...  Done
    Checking conf files for problems...
            Improper stanza [dhcpd_server_dhcprelease] in /opt/splunk/etc/ap                                                                                                                                                                                                                                             ps/unix/default/tags.conf, line 30
            Your indexes and inputs configurations are not internally consis                                                                                                                                                                                                                                             tent. For more information, run 'splunk btool check --debug'
    Done
    Checking default conf files for edits...
    Validating installed files against hashes from '/opt/splunk/splunk-6.6.3                                                                                                                                                                                                                                             -e21ee54bc796-linux-2.6-x86_64-manifest'
    All installed files intact.
    Done

All preliminary checks passed.

Starting splunk server daemon (splunkd)...
Done
[ OK ]

Waiting for web server at https://10.244.161.7:8000 to be available... Done

If you get stuck, we're here to help.
Look for answers here: http://docs.splunk.com

The Splunk web interface is at https://10.244.161.7:8000

0 Karma

SplunkTrust
SplunkTrust

Splunk started perfectly fine, only one warning message because you are using old version of Splunk App for Unix and Linux. Please upgrade that, you might need to remove old unix app because new app folder name has been chanegd to splunkappfor_nix

0 Karma

New Member

After i log with my credentials i get an "500 Internal Error" doesnt matter the username.
alt text

0 Karma

SplunkTrust
SplunkTrust

Any error in $SPLUNKHOME/var/log/splunk/webservice.log ?

0 Karma

New Member

Yes
link text

0 Karma

New Member
0 Karma

New Member
0 Karma

SplunkTrust
SplunkTrust

Looks like some cherrypy session related problem , I'll suggest to open case with splunk support.

0 Karma

Builder

Is https still being used after the upgrade? Try http - it could have reset or not loading conf files properly. Or is this message only occurring when trying to load a certain page?

0 Karma

New Member

http doesnt work, only https.
The error occur after i login with my credentials.
I dont know what to do.

*> # ./splunk start

Splunk> Now with more code!

Checking prerequisites...
Checking http port [10.244.161.7:8000]: open
Checking mgmt port [10.244.161.7:8089]: open
Checking appserver port [127.0.0.1:8065]: open
Checking kvstore port [10.244.161.7:8191]: open
Checking configuration... Done.
Checking critical directories... Done
Checking indexes...
Validated: audit _internal _introspection _telemetry _thefishbu cket checkfwd eqalisnetworksample
firewall history itau main mwg
audit
os osse

c perfmon snortcardholder
snort
servidores sos sossummarydaily
summary summar

yforwarders summaryhosts
summaryindexers summarypools
summarysources summar

y
sourcetypes syslog tpwinsec
tpwinservers windows wineventlog
Done

Bypassing local license checks since
this instance is configured with a
remote l

icense master.

    Checking filesystem compatibility...  Done
    Checking conf files for problems...
            Improper stanza [dhcpd_server_dhcprelease] in

/opt/splunk/etc/ap

ps/unix/default/tags.conf, line 30
Invalid key in stanza [email] in
/opt/splunk/etc/system/local/al

ertactions.conf, line 5:
reportServerEnabled (value: 1).
Invalid key in stanza [email] in
/opt/splunk/etc/system/local/al

ert
actions.conf, line 6:
reportServerURL (value: ).
Your indexes and inputs configurations are not
internally consis

tent. For more information, run
'splunk btool check --debug'
Done
Checking default conf files for edits...
Validating installed files against hashes from
'/opt/splunk/splunk-6.6.3

-e21ee54bc796-linux-2.6-x86_64-manifest'
All installed files intact.
Done All preliminary checks passed.

Starting splunk server daemon
(splunkd)... Done
[ OK ]

Waiting for web server at
https://10.244.161.7:8000 to be
available... Done

If you get stuck, we're here to help.
Look for answers here:
http://docs.splunk.com

The Splunk web interface is at
https://10.244.161.7:8000*

0 Karma

Builder

Are other users experiencing the same problem or just you? It could be ssl related. Are you running in a distributed environment, can you log in to other servers?

I can see you have a few conf errors as well - this link should help you fix those old values for the conf files:
https://answers.splunk.com/answers/548915/after-a-successful-upgrade-from-621-to-661-we-are.html

0 Karma

New Member

All users on the same splunk indexer are experiencing the error, the others with older enterprise version have no problem.
What can be causing it? The server only runs splunk enterprise.

0 Karma