When trying to start Splunk, why am I getting "WARNING: web interface does not seem to be available!"?

[ali_alnajjar_ve]

Hi Guys:

Please who can give me a help !!
I'm not able to start splunk.

bash-4.1$ /opt/splunk/bin/splunk start

Splunk> Be an IT superhero. Go home early.

Checking prerequisites...
        Checking http port [8000]: open
        Checking mgmt port [8089]: open
        Checking appserver port [127.0.0.1:8065]: open
        Checking kvstore port [8191]: open
        Checking configuration...  Done.
        Checking critical directories...        Done
        Checking indexes...
                Validated: _audit _internal _introspection _thefishbucket access_summary access_summary2 algosec audit_summary audit_summary2 bcoat bro cim_summary cisco_acs cisco_ise cisco_router cisco_wc endpoint_summary endpoint_summary2 firedalerts fireeye guardium history ioc juniper_isg main mcafee_eg mcafee_ips misc network_summary network_summary2 network_summary3 nexthink notable notable_summary oim os os_aix os_hpux os_linux os_sunos os_windows paloalto_pa perfmon proxy_center_summary proxy_center_summary2 qualys risk rsa_ecat rsa_sa session_end session_start summary symantec_dlp symantec_encryption symantec_sep te test threat_activity tpam traffic_center_summary traffic_center_summary2 ueba vasco venafi web_inspect websense whois windows wineventlog wrla xtreme_contexts
        Done


Bypassing local license checks since this instance is configured with a remote license master.

    Checking filesystem compatibility...  Done
    Checking conf files for problems...
    Done
    Checking default conf files for edits...
    Validating installed files against hashes from '/opt/splunk/splunk-6.3.2-aaff59bb082c-linux-2.6-x86_64-manifest'
    All installed files intact.
    Done
    Checking replication_port port [9887]: open
All preliminary checks passed.

Starting splunk server daemon (splunkd)...
Done
                                                           [  OK  ]

Waiting for web server at http://127.0.0.1:8000 to be available.splunkd 3600 was not running.
Stopping splunk helpers...
                                                           [  OK  ]
Done.
Stopped helpers.
Removing stale pid file... done.


WARNING: web interface does not seem to be available!

[plarsenDST]

Indexer seemed to work fine, no web interface no 8000 listening port.

Our issue was these files were corrupted likely by cisco AMP.

-rw-------. 1 splunk splunk 5165 Nov 20 12:55 times.pyo
-rw-------. 1 splunk splunk 13008 Nov 20 12:55 routes.pyo
-rw-------. 1 splunk splunk 15667 Nov 20 12:55 message.pyo
-rw-------. 1 splunk splunk 8204 Nov 20 12:56 startup.pyo

/apps/splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle/lib

Copied from other index server and restarted, web interface was available again.

[anand_singh17]

Please look for Splunkd.log.

Observation:
1. bucket replication or bucket duplication
2. Indexes execution issue (if cluster enviornment)

Resolution
1. check carefully entire (try to find for "error", you may get duplicate buckets.
2. In case of indexes, you may not have your master cluster speaking, or bucket under other indexes may be creating this issue.

Regards,
Anand

[hortonew]

I just had this issue with bucket conflict. Look for the following in splunkd.log:

Error IndexerService - Error initilizing IndexerService: idx=* bucket=rb_* Detected directory manually copied into its database, causing id conflics ...

[katanguriabhi]

How did you solved your problem. I am experiencing the same.

[hortonew]

Got rid of the conflicts - since mine were replicated buckets I just deleted them. You could rename yours if need be.

[katanguriabhi]

even mine were replicated buckets and i deleted them and tried restarting but at last i am getting the error.
10-22-2016 14:27:19.243 -0700 FATAL IndexerService - One or more indexes could not be initialized. Cannot disable indexes on a clustering slave.

[ali_alnajjar_ve · Accepted solution]

Solved,

thanks folks.

just added the indexer to cluster master through web instead of direct editing config files.

[asabatini85]

Hi

can I ask which file you pushed?

BR

[Raghav2384]

Several reasons that can cause this issue. without looking at splunkd.log, it's hard to tell

i can list few possibilities from my experience

if it's an indexer,
1. see if there are any bucket clashes....if there are two buckets with same id, example db__120 in db and db__120 in colddb

if it's a search head,
1. if you pushed an encrypted password file from deployer, shc member fail to parse the file as it doesn't know what the encrypted password is.

Again, hard to tell without seeing the splunkd.log.

Hope this helps!

Thanks,
Raghav

[maciep]

Anything in the splunkd logs on that sever? Are you using ssl? Create your own certs?