Why am I getting an internal server error at login?

[Parv007]

I just installed Splunk and when started it in the browser, first time signing in option was there with admin and changeme. I entered it and it navigated me to the next page to set up a new password. Once I entered the new password and clicked on set password, I was taken to a page saying:

500 Internal Server Error
Return to Splunk home page

View more information about your request (request ID = 5acb8c24f81a8c0e3b2e8) in Search
This page was linked to from http://localhost:8000/.
You are using localhost:8000, which is connected to splunkd @000 at https://127.0.0.1:8089 on Mon Apr 9 21:22:05 2018.

Further checked splunkd.log . These are the errors :
04-10-2018 06:59:34.801 +0530 ERROR ExecProcessor - message from "python "C:\Program Files\Splunk\etc\apps\splunk_instrumentation\bin\on_splunk_start.py"" ERROR:InstrumentationInit:[Errno 10054] An existing connection was forcibly closed by the remote host
04-10-2018 06:59:35.311 +0530 INFO IntrospectionGenerator:resource_usage - RU_main - I-data gathering (Resource Usage) starting; period=10s
04-10-2018 06:59:35.311 +0530 WARN IntrospectionGenerator:resource_usage - RU - Failure executing initial disk PDH query, status code is -2147481643
04-10-2018 06:59:35.311 +0530 INFO IntrospectionGenerator:resource_usage - RU_main - I-data gathering (IO Statistics) starting; interval=60s
04-10-2018 06:59:35.311 +0530 WARN IntrospectionGenerator:resource_usage - RU - Failure executing PDH query, skipping getting iostats data this collection cycle. Status code is -2147481643
04-10-2018 06:59:36.819 +0530 ERROR ExecProcessor - message from "python "C:\Program Files\Splunk\etc\apps\splunk_monitoring_console\bin\dmc_config.py"" Socket error communicating with splunkd (error=[Errno 10054] An existing connection was forcibly closed by the remote host), path = /services/shcluster/config?output_mode=json
04-10-2018 06:59:36.819 +0530 INFO ExecProcessor - message from "python "C:\Program Files\Splunk\etc\apps\splunk_monitoring_console\bin\dmc_config.py"" Cannot connect to splunkd. Will not disable DMC.
04-10-2018 07:00:16.733 +0530 ERROR SearchOperator:copyresults - Cannot find results for search_id 'scheduler_nobody_c3BsdW5rX21vbml0b3JpbmdfY29uc29sZQRMD54740dfff07b17ef1_at_1523323765_0'.
04-10-2018 07:00:16.733 +0530 ERROR SearchScheduler - Error in 'SearchOperator:copyresults': Cannot find results for search_id 'schedulernobody_c3BsdW5rX21vbml0b3JpbmdfY29uc29sZQRMD54740dfff07b17ef1_at_1523323765_0'., search='copyresults dest="dmc_assets" sid="schedulernobody_c3BsdW5rX21vbml0b3JpbmdfY29uc29sZQ_RMD54740dfff07b17ef1_at_1523323765_0"'
04-10-2018 07:00:27.438 +0530 INFO DatabaseDirectoryManager - idx=_internal Writing a bucket manifest in hotWarmPath='C:\Program Files\Splunk\var\lib\splunk_internaldb\db', pendingBucketUpdates=0 . Reason='Buckets were rebuilt or tsidx-minified (bucket_count=1).'

How to resolve this issue?

[p_gurav]

Be sure to set the following before starting for the first time:

export SPLUNK_HOME=/my/splunk/dir
source $SPLUNK_HOME/bin/setSplunkEnv
Add $SPLUNK_HOME/bin to your PATH
export PATH=$SPLUNK_HOME/bin:$PATH

Check to make sure 'which python' is referencing the local, splunk, python libs:

[splunkysploo]# which python
/<$SPLUNK_HOME>/bin/python

Then, start Splunk with $SPLUNK_HOME/bin/splunk start as per usual.

[Parv007]

I am using it on windows and not Linux . Can you share the steps to rectify this issue on windows .

[p_gurav]

This may help:
http://dev.splunk.com/view/quickstart/SP-CAAAFDH

Also check that 8000 port is open on your system.

[MuS]

check splunkd.log ( $SPLUNK_HOME\var\log\splunk\ ) to see what the error is about, also check if any firewall or Windows UAC is playing badly.

cheers, MuS

[Parv007]

Please see . I have updated the issue . Help me in rectifying those errors which are listed above .

[Parv007]

04-10-2018 06:59:34.801 +0530 ERROR ExecProcessor - message from "python "C:\Program Files\Splunk\etc\apps\splunk_instrumentation\bin\on_splunk_start.py"" ERROR:InstrumentationInit:[Errno 10054] An existing connection was forcibly closed by the remote host
04-10-2018 06:59:35.311 +0530 WARN IntrospectionGenerator:resource_usage - RU - Failure executing initial disk PDH query, status code is -2147481643
04-10-2018 06:59:35.311 +0530 WARN IntrospectionGenerator:resource_usage - RU - Failure executing PDH query, skipping getting iostats data this collection cycle. Status code is -2147481643
04-10-2018 06:59:36.819 +0530 ERROR ExecProcessor - message from "python "C:\Program Files\Splunk\etc\apps\splunk_monitoring_console\bin\dmc_config.py"" Socket error communicating with splunkd (error=[Errno 10054] An existing connection was forcibly closed by the remote host), path = /services/shcluster/config?output_mode=json
04-10-2018 06:59:36.819 +0530 INFO ExecProcessor - message from "python "C:\Program Files\Splunk\etc\apps\splunk_monitoring_console\bin\dmc_config.py"" Cannot connect to splunkd. Will not disable DMC.
04-10-2018 07:00:16.733 +0530 ERROR SearchOperator:copyresults - Cannot find results for search_id 'scheduler_nobody_c3BsdW5rX21vbml0b3JpbmdfY29uc29sZQRMD54740dfff07b17ef1_at_1523323765_0'.
04-10-2018 07:00:16.733 +0530 ERROR SearchScheduler - Error in 'SearchOperator:copyresults': Cannot find results for search_id 'schedulernobody_c3BsdW5rX21vbml0b3JpbmdfY29uc29sZQRMD54740dfff07b17ef1_at_1523323765_0'., search='copyresults dest="dmc_assets" sid="schedulernobody_c3BsdW5rX21vbml0b3JpbmdfY29uc29sZQ_RMD54740dfff07b17ef1_at_1523323765_0"'
04-10-2018 07:00:27.438 +0530 INFO DatabaseDirectoryManager - idx=_internal Writing a bucket manifest in hotWarmPath='C:\Program Files\Splunk\var\lib\splunk_internaldb\db', pendingBucketUpdates=0 . Reason='Buckets were rebuilt or tsidx-minified (bucket_count=1).'