The Events Service cluster may not function correctly and its health state will be listed as red if there are unwanted indices either in the unassigned or initializing state.
Example:
In a browser or terminal, run the following:
curl http://localhost:9081/healthcheck?pretty=true
The output of the healthcheck command will show cluster state [RED]
.
"events-service-api-store / elasticsearch-singlenode-module" : { "healthy" : true, "message" : "Current [appdynamics-events-service-cluster] cluster state: [RED], data nodes: [1], nodes: [1], active shards: [48], relocating shards: [0], initializing shards: [0], unassigned shards: [5], timed out: [false]" },
These unwanted indices are created as a result of the Nessus Vulnerability Scanner that gets run on the cluster port (default 9200).
Run the following command on the Events Service machine or in a browser to verify if there are unwanted indices.
http://<events_service_machine>:9200/_cat/shards?v [in Browser] OR curl http://<events_service_machine>:9200/_cat/shards?v [in the terminal of the Events Service machine]
Output example:
index shard prirep state docs store ip node vrvsgs1.html 4 p STARTED 0 114b 192.x.y.z node-xyzxyzxyz vrvsgs1.html 4 r UNASSIGNED 0 168b 192.x.y.z node-xyzxyzxyz vrvsgs1.html 0 r UNASSIGNED 0 791b 192.x.y.z node-xyzxyzxyz vrvsgs1.html 0 p STARTED 0 125b 192.x.y.z node-xyzxyzxyz vrvsgs1.html 3 r STARTED 0 105b 192.x.y.z node-xyzxyzxyz gvanws1.html 4 p STARTED 0 125b 192.x.y.z node-xyzxyzxyz gvanws1.html 4 r STARTED 0 128b 192.x.y.z node-xyzxyzxyz gvanws1.html 0 p STARTED 0 115b 192.x.y.z node-xyzxyzxyz gvanws1.html 0 r STARTED 0 144b 192.x.y.z node-xyzxyzxyz gvanws1.html 3 p STARTED 0 115b 192.x.y.z node-xyzxyzxyz oapssw1.html 4 p STARTED 0 115b 192.x.y.z node-xyzxyzxyz oapssw1.html 0 r STARTED 0 144b 192.x.y.z node-xyzxyzxyz oapssw1.html 0 p STARTED 0 115b 192.x.y.z node-xyzxyzxyz nrawos1.html 2 p STARTED 0 115b 192.x.y.z node-xyzxyzxyz nrawos1.html 2 r STARTED 0 108b 192.x.y.z node-xyzxyzxyz nrawos1.html 0 p STARTED 0 115b 192.x.y.z node-xyzxyzxyz nrawos1.html 0 r STARTED 0 108b 192.x.y.z node-xyzxyzxyz spipe 4 p STARTED 0 115b 192.x.y.z node-xyzxyzxyz spipe 4 r STARTED 0 144b 192.x.y.z node-xyzxyzxyz spipe 0 p STARTED 0 115b 192.x.y.z node-xyzxyzxyz spipe 0 r STARTED 0 144b 192.x.y.z node-xyzxyzxyz psaoon1.html 2 p STARTED 0 115b 192.x.y.z node-xyzxyzxyz psaoon1.html 2 r STARTED 0 108b 192.x.y.z node-xyzxyzxyz psaoon1.html 0 p STARTED 0 115b 192.x.y.z node-xyzxyzxyz psaoon1.html 0 r STARTED 0 144b 192.x.y.z node-xyzxyzxyz arenon1.html 4 r STARTED 0 144b 192.x.y.z node-xyzxyzxyz arenon1.html 0 r STARTED 0 144b 192.x.y.z node-xyzxyzxyz arenon1.html 0 p STARTED 0 115b 192.x.y.z node-xyzxyzxyz arenon1.html 3 r STARTED 0 144b 192.x.y.z node-xyzxyzxyz sprreo1.html 1 r STARTED 0 144b 192.x.y.z node-xyzxyzxyz sprreo1.html 2 r STARTED 0 144b 192.x.y.z node-xyzxyzxyz sprreo1.html 2 p UNASSIGNED 0 115b 192.x.y.z node-xyzxyzxyz vepper1.html 4 p STARTED 0 115b 192.x.y.z node-xyzxyzxyz vepper1.html 4 r STARTED 0 144b 192.x.y.z node-xyzxyzxyz vepper1.html 0 p STARTED 0 115b 192.x.y.z node-xyzxyzxyz perl 1 r STARTED 0 144b 192.x.y.z node-xyzxyzxyz perl 2 r STARTED 0 144b 192.x.y.z node-xyzxyzxyz perl 2 p STARTED 0 115b 192.x.y.z node-xyzxyzxyz gaowsa1.html 4 p UNASSIGNED 0 115b 192.x.y.z node-xyzxyzxyz greprs1.html 4 r STARTED 0 144b 192.x.y.z node-xyzxyzxyz greprs1.html 4 p STARTED 0 115b 192.x.y.z node-xyzxyzxyz enoson1.html 4 p STARTED 0 115b 192.x.y.z node-xyzxyzxyz enoson1.html 4 r STARTED 0 108b 192.x.y.z node-xyzxyzxyz enoson1.html 0 p STARTED 0 115b 192.x.y.z node-xyzxyzxyz vpnwwg1.html 2 r STARTED 0 144b 192.x.y.z node-xyzxyzxyz vpnwwg1.html 2 p STARTED 0 115b 192.x.y.z node-xyzxyzxyz arenon1.html 4 p STARTED 0 115b 192.x.y.z node-xyzxyzxyz arenon1.html 4 r STARTED 0 144b 192.x.y.z node-xyzxyzxyz soaeap1.html 3 p STARTED 0 115b 192.x.y.z node-xyzxyzxyz soaeap1.html 1 r STARTED 0 144b 192.x.y.z node-xyzxyzxyz pwvwgg1.html 1 p STARTED 0 115b 192.x.y.z node-xyzxyzxyz pwvwgg1.html 2 p STARTED 0 115b 192.x.y.z node-xyzxyzxyz pwvwgg1.html 2 r STARTED 0 144b 192.x.y.z node-xyzxyzxyz enawae1.html 4 r STARTED 0 108b 192.x.y.z node-xyzxyzxyz enawae1.html 4 p UNASSIGNED 0 115b 192.x.y.z node-xyzxyzxyz sawmill6cl.exe 4 p UNASSIGNED 0 115b 192.x.y.z node-xyzxyzxyz sawmill6cl.exe 4 r STARTED 0 108b 192.x.y.z node-xyzxyzxyz rogwsw1.html 4 r STARTED 0 108b 192.x.y.z node-xyzxyzxyz
Delete these unwanted indices using the following commands:
curl -XDELETE http://<events-service_hostname>:9200/<index_name>
curl -XDELETE http://localhost:9200/rogwsw1.html
curl http://<events_service_machine>:9200/_cat/shards?v
curl http://<events_service_machine>:9200/_cat/indices?v