Hi there,
Thanks for the output and it's very clear now about why we are seeing the below messages repeatedly about (all shards failed) in your server logs.
Log Snippet:
#|2017-05-02T12:18:05.795+0500|SEVERE|glassfish 4.1|com.singularity.ee.controller.beans.ExceptionHandlingInterceptor|_ThreadID=307;_ThreadName=http-listener-1(5);_TimeMillis=1493709485795;_LevelValue=1000;|Encountered runtime exception
RestException(statusCode=503, code=Service.Unavailable, errorMessage=Service unavailable: [all shards failed], developerMessage=Exception Message Chain : Failed to execute phase [query_fetch], all shards failed
Caused/related: all shards failed
From the healthcheck output,I infer your cluster state is in RED state due to unassigned shards and hence the above error which says "all shards failed" as a result of it.
"events-service-api-store / elasticsearch-singlenode-module" : {
"healthy" : false,
"message" : "Current [appdynamics-events-service-cluster] cluster state: [RED], data nodes: [1], nodes: [1], active shards: [2], relocating shards: [0], initializing shards: [0], unassigned shards: [18], timed out: [false]"
},
Below Shards output shows which indices are in UNASSIGNED state and all the dbmon event type indices are in UNASSIGNED
[dynamics@app-dyn ~]$ curl http://localhost:9200/_cat/shards?v
index shard prirep state docs store ip node
async_executor_status 0 p UNASSIGNED
appdynamics_version 0 p UNASSIGNED
biz_outcome_definitions 0 p UNASSIGNED
event_type_extracted_fields 0 p UNASSIGNED
appdynamics_meters_v2 0 p UNASSIGNED
event_type_metadata_hidden_fields_v1 0 p UNASSIGNED
appdynamics_entity_metadata 0 p STARTED 0 159b node-2c7781cf-58e5-4784-b11c-5fc6272af932
customer1_39714636-eec0-4331-932d-1394d05fabd3___dbmon-query-stat___2017-04-27_12-56-07 1 p UNASSIGNED
customer1_39714636-eec0-4331-932d-1394d05fabd3___dbmon-query-stat___2017-04-27_12-56-07 2 p UNASSIGNED
customer1_39714636-eec0-4331-932d-1394d05fabd3___dbmon-query-stat___2017-04-27_12-56-07 0 p UNASSIGNED
customer1_39714636-eec0-4331-932d-1394d05fabd3___dbmon-wait-time___2017-04-27_12-56-05 1 p UNASSIGNED
customer1_39714636-eec0-4331-932d-1394d05fabd3___dbmon-wait-time___2017-04-27_12-56-05 2 p UNASSIGNED
customer1_39714636-eec0-4331-932d-1394d05fabd3___dbmon-wait-time___2017-04-27_12-56-05 0 p UNASSIGNED
customer1_39714636-eec0-4331-932d-1394d05fabd3___dbmon-execution-plan-details___2017-04-27_12-56-08 1 p UNASSIGNED
customer1_39714636-eec0-4331-932d-1394d05fabd3___dbmon-execution-plan-details___2017-04-27_12-56-08 2 p UNASSIGNED
customer1_39714636-eec0-4331-932d-1394d05fabd3___dbmon-execution-plan-details___2017-04-27_12-56-08 0 p UNASSIGNED
slm_performance_configs_v1 0 p UNASSIGNED
appdynamics_accounts 0 p STARTED 2 17.6kb node-2c7781cf-58e5-4784-b11c-5fc6272af932
event_type_metadata 0 p UNASSIGNED
appdynamics_api_keys_v1 0 p UNASSIGNED
This can happen for a number of reasons and hence to understand the reason behind it, Please execute the following command and share its output
curl -s -XGET http://localhost:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason | grep UNASSIGNED
Regards,
Mohammed Rayan
... View more