Unable to connect to the splunk web interface

[cebo_myeza]

I have been using the splunk web interface with this address 127.0.0.1:8000 for almost two months now, but all of the sudden it just stopped connecting and it's giving me a message saying that "Firefox can't establish a connection to the server at 127.0.0.1:8000".

please help i don't know what went wrong.

[cebo_myeza]

Thanks Mus

please advise me what to do from this point far because i feel like i am stacked 😞

thanks

[MuS]

good, at least it looks like the files are still around....

[MuS]

What happens if your run:

strace /opt/splunk/bin/splunk start

[n00badmin]

can you check your splunkd.log please

cd /opt/splunk/var/log/splunk/

tail -100 splunkd.log

dont understand why you get nothing calling ./splunk start or stop

[cebo_myeza]

[root@localhost wisdom.network_trainee]# cd /opt/splunk/var/log/splunk/

[root@localhost splunk]# tail -50 splunkd.log
06-18-2015 10:40:47.859 +0700 INFO DatabaseDirectoryManager - Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/_internaldb/db'. Reason='Removing bucket, bid=_internal~26~414C0929-B95F-4C97-A4BB-AB1267F53365'
06-18-2015 10:40:47.890 +0700 INFO BucketMover - AsyncFreezer freeze succeeded for bkt='/opt/splunk/var/lib/splunk/_internaldb/db/db_1432006804_1431942980_26'
06-18-2015 10:40:48.839 +0700 INFO DatabaseDirectoryManager - Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/_internaldb/db'. Reason=' frozen_buckets'
06-18-2015 10:41:44.726 +0700 INFO WatchedFile - Will begin reading at offset=0 for file='/opt/splunk/var/log/splunk/metrics.log'.
06-18-2015 10:41:44.864 +0700 INFO WatchedFile - Will begin reading at offset=24992205 for file='/opt/splunk/var/log/splunk/metrics.log.1'.
06-18-2015 10:42:48.840 +0700 INFO BucketMover - will attempt to freeze: candidate='/opt/splunk/var/lib/splunk/_internaldb/db/db_1432006922_1432006809_27' because frozenTimePeriodInSecs=2592000 exceeds difference between now=1434598968 and latest=1432006922
06-18-2015 10:42:48.843 +0700 INFO DatabaseDirectoryManager - Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/_internaldb/db'. Reason='Removing bucket, bid=_internal~27~414C0929-B95F-4C97-A4BB-AB1267F53365'
06-18-2015 10:42:48.847 +0700 INFO BucketMover - AsyncFreezer freeze succeeded for bkt='/opt/splunk/var/lib/splunk/_internaldb/db/db_1432006922_1432006809_27'
06-18-2015 10:42:49.832 +0700 INFO DatabaseDirectoryManager - Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/_internaldb/db'. Reason=' frozen_buckets'
06-18-2015 15:16:56.840 +0700 INFO BucketMover - will attempt to freeze: candidate='/opt/splunk/var/lib/splunk/_internaldb/db/db_1432023396_1432006926_28' because frozenTimePeriodInSecs=2592000 exceeds difference between now=1434615416 and latest=1432023396
06-18-2015 15:16:56.841 +0700 INFO DatabaseDirectoryManager - Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/_internaldb/db'. Reason='Removing bucket, bid=_internal~28~414C0929-B95F-4C97-A4BB-AB1267F53365'
06-18-2015 15:16:56.867 +0700 INFO BucketMover - AsyncFreezer freeze succeeded for bkt='/opt/splunk/var/lib/splunk/_internaldb/db/db_1432023396_1432006926_28'
06-18-2015 15:16:57.832 +0700 INFO DatabaseDirectoryManager - Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/_internaldb/db'. Reason=' frozen_buckets'
06-18-2015 17:03:47.839 +0700 INFO BucketMover - will attempt to freeze: candidate='/opt/splunk/var/lib/splunk/_introspection/db/db_1433412195_1433380011_38' because frozenTimePeriodInSecs=1209600 exceeds difference between now=1434621827 and latest=1433412195
06-18-2015 17:03:47.842 +0700 INFO DatabaseDirectoryManager - Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/_introspection/db'. Reason='Removing bucket, bid=_introspection~38~414C0929-B95F-4C97-A4BB-AB1267F53365'
06-18-2015 17:03:47.862 +0700 INFO BucketMover - AsyncFreezer freeze succeeded for bkt='/opt/splunk/var/lib/splunk/_introspection/db/db_1433412195_1433380011_38'
06-18-2015 17:03:58.837 +0700 INFO DatabaseDirectoryManager - Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/_introspection/db'. Reason=' frozen_buckets'
06-18-2015 17:08:36.835 +0700 INFO BucketMover - will attempt to freeze: candidate='/opt/splunk/var/lib/splunk/_internaldb/db/db_1432030087_1432023400_29' because frozenTimePeriodInSecs=2592000 exceeds difference between now=1434622116 and latest=1432030087
06-18-2015 17:08:36.835 +0700 INFO DatabaseDirectoryManager - Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/_internaldb/db'. Reason='Removing bucket, bid=_internal~29~414C0929-B95F-4C97-A4BB-AB1267F53365'
06-18-2015 17:08:36.844 +0700 INFO BucketMover - AsyncFreezer freeze succeeded for bkt='/opt/splunk/var/lib/splunk/_internaldb/db/db_1432030087_1432023400_29'
06-18-2015 17:08:37.831 +0700 INFO DatabaseDirectoryManager - Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/_internaldb/db'. Reason=' frozen_buckets'
06-19-2015 00:00:00.831 +0700 INFO LMStackMgr - should rollover=true because _lastRolloverTime=1434560400 lastRolloverDay=1434560400 snappedNow=1434646800
06-19-2015 00:00:00.832 +0700 INFO LMStackMgr - quotaExceededCount=0, lastExceedDate=0, peak=0, rolloverCount=1, totalCumulativeBytesAtRollover=0, todaysBytesIndexed=0, licenseSize=524288000
06-19-2015 00:00:00.832 +0700 INFO LMStackMgr - finished rollover, new lastRolloverTime=1434646800
06-19-2015 00:00:03.831 +0700 INFO LMSlaveInfo - Detected that masterTimeFromSlave(Thu Jun 18 23:59:02 2015) < lastRolloverTime(Fri Jun 19 00:00:00 2015), meaning that the master has already rolled over. Ignore slave persisted usage.
06-19-2015 07:36:27.586 +0700 INFO WatchedFile - Will begin reading at offset=0 for file='/opt/splunk/var/log/introspection/resource_usage.log'.
06-19-2015 07:36:27.668 +0700 INFO WatchedFile - Will begin reading at offset=24998813 for file='/opt/splunk/var/log/introspection/resource_usage.log.1'.
06-19-2015 09:24:41.898 +0700 INFO WatchedFile - Will begin reading at offset=0 for file='/opt/splunk/var/log/splunk/metrics.log'.
06-19-2015 09:24:42.020 +0700 INFO WatchedFile - Will begin reading at offset=24997732 for file='/opt/splunk/var/log/splunk/metrics.log.1'.
06-19-2015 10:42:14.568 +0700 ERROR HTTPClient - Cannot resolve IP of host=apps.splunk.com: Temporary failure in name resolution
06-19-2015 10:42:14.568 +0700 ERROR ApplicationUpdater - Error checking for update, URL=/api/apps:resolve/checkforupgrade: Invalid URI
06-19-2015 10:51:20.832 +0700 INFO LMStackMgr - should rollover=true because _lastRolloverTime=1434646800 lastRolloverDay=1434560400 snappedNow=1434646800
06-19-2015 10:51:20.832 +0700 INFO LMStackMgr - quotaExceededCount=0, lastExceedDate=0, peak=0, rolloverCount=2, totalCumulativeBytesAtRollover=0, todaysBytesIndexed=0, licenseSize=524288000
06-19-2015 10:51:20.832 +0700 INFO LMStackMgr - finished rollover, new lastRolloverTime=1434685880
06-19-2015 10:51:48.586 +0700 WARN TimeoutHeap - Detected system time adjusted backwards by 11245ms.
06-19-2015 10:51:48.663 +0700 WARN TimeoutHeap - Detected system time adjusted backwards by 11244ms.
06-19-2015 10:51:48.747 +0700 WARN TimeoutHeap - Detected system time adjusted backwards by 11244ms.
06-19-2015 10:51:48.765 +0700 WARN TimeoutHeap - Detected system time adjusted backwards by 2326ms.
06-19-2015 10:51:48.908 +0700 WARN IntrospectionGenerator:resource_usage - TimeoutHeap - Detected system time adjusted backwards by 7247ms.
06-19-2015 10:51:48.908 +0700 WARN TimeoutHeap - Detected system time adjusted backwards by 7247ms.
06-19-2015 10:51:52.849 +0700 WARN TimeoutHeap - Detected system time adjusted backwards by 2246ms.
06-19-2015 10:51:52.849 +0700 WARN TimeoutHeap - Detected system time adjusted backwards by 2246ms.
06-19-2015 10:52:16.584 +0700 INFO LMSlaveInfo - Detected that masterTimeFromSlave(Fri Jun 19 10:51:15 2015) < lastRolloverTime(Fri Jun 19 10:51:20 2015), meaning that the master has already rolled over. Ignore slave persisted usage.
06-19-2015 13:41:17.605 +0700 INFO BucketMover - will attempt to freeze: candidate='/opt/splunk/var/lib/splunk/_internaldb/db/db_1432104061_1432030091_30' because frozenTimePeriodInSecs=2592000 exceeds difference between now=1434696077 and latest=1432104061
06-19-2015 13:41:17.606 +0700 INFO DatabaseDirectoryManager - Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/_internaldb/db'. Reason='Removing bucket, bid=_internal~30~414C0929-B95F-4C97-A4BB-AB1267F53365'
06-19-2015 13:41:17.629 +0700 INFO BucketMover - AsyncFreezer freeze succeeded for bkt='/opt/splunk/var/lib/splunk/_internaldb/db/db_1432104061_1432030091_30'
06-19-2015 13:41:18.585 +0700 INFO DatabaseDirectoryManager - Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/_internaldb/db'. Reason=' frozen_buckets'
06-19-2015 16:50:44.599 +0700 INFO BucketMover - will attempt to freeze: candidate='/opt/splunk/var/lib/splunk/_introspection/db/db_1433497801_1433466532_39' because frozenTimePeriodInSecs=1209600 exceeds difference between now=1434707444 and latest=1433497801
06-19-2015 16:50:44.605 +0700 INFO DatabaseDirectoryManager - Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/_introspection/db'. Reason='Removing bucket, bid=_introspection~39~414C0929-B95F-4C97-A4BB-AB1267F53365'
06-19-2015 16:50:44.608 +0700 INFO BucketMover - AsyncFreezer freeze succeeded for bkt='/opt/splunk/var/lib/splunk/_introspection/db/db_1433497801_1433466532_39'

[n00badmin]

is ur system healthy?

df -h

your partitions still have space??

[cebo_myeza]

yeah this was the last time

06-19-2015 16:50:44.608 +0700 INFO BucketMover - AsyncFreezer freeze succeeded for bkt='/opt/splunk/var/lib/splunk/_introspection/db/db_1433497801_1433466532_39'

after this i was not able to enter splunk web

[n00badmin]

is that ur last log?

at 16:50:44 on 06-19??

[n00badmin]

i doubt u were able to post the entire 100 lines of log...

do a tail -10 please...whats ur last 10 events in the log?

[esix_splunk]

Try restarting your instance of splunk : 'sudo /opt/splunk/bin/splunk restart'. Check the message when it restarts and confirm what IP and port Splunk is bound to. (It will say started on http://localhost:8000 or https://localhost:XXXX"

Check if iptables has been started or running : service iptables status service iptables stop

[cebo_myeza]

hi esix_splunk

The iptables seems ok but when i run the command 'sudo /opt/splunk/bin/splunk restart' i don't get any response.

i get this:

[root@...]# sudo /opt/splunk/bin/splunk restart
[root@..]#

[martin_mueller]

Assuming Splunk is running, someone may have changed the port or HTTPS encryption.

[n00badmin]

is splunk running?

what OS are you on?

[cebo_myeza]

hi n00badmin

i am using centos and splunk is running