topic Re: No query data in Databasa statistics in Splunk AppDynamics

No query data in Databasa statistics

CommunityUser — Tue, 25 Apr 2017 09:33:19 GMT

Hello!

We are making POC of AppDynamics with our Customer and we've met the following issue. The dbagent shows Wait Time graph in AppDynamics GUI but shows no SQL queries and "Internal Error 500" when we try to open Queries tab. In agent.log we see the following error:

25 Apr 2017 14:40:05,756 WARN [Event-Uploader-Scheduler-2] DBMeasurementUploader:81 - Fail to upload Wait Time Measurements:
com.singularity.ee.rest.ResponseReadException: Error in controller in processing binary request Wait Time Measurement - null
        at com.singularity.ee.rest.controller.request.ABinaryControllerRequest.<init>(ABinaryControllerRequest.java:65)
        at com.singularity.ee.rest.controller.request.dbmon.WaitTimeMeasurementCollectorRequest.<init>(WaitTimeMeasurementCollectorRequest.java:34)
        at com.singularity.ee.agent.dbagent.task.reporter.DBMeasurementUploader.uploadWaitTimeMeasurements(DBMeasurementUploader.java:79)
        at com.singularity.ee.agent.dbagent.task.reporter.DBMeasurementUploader.run(DBMeasurementUploader.java:133)
        at com.singularity.ee.util.javaspecific.scheduler.AgentScheduledExecutorServiceImpl$SafeRunnable.run(AgentScheduledExecutorServiceImpl.java:122)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at com.singularity.ee.util.javaspecific.scheduler.ADFutureTask$Sync.innerRunAndReset(ADFutureTask.java:335)
        at com.singularity.ee.util.javaspecific.scheduler.ADFutureTask.runAndReset(ADFutureTask.java:152)
        at com.singularity.ee.util.javaspecific.scheduler.ADScheduledThreadPoolExecutor$ADScheduledFutureTask.access$101(ADScheduledThreadPoolExecutor.java:119)
        at com.singularity.ee.util.javaspecific.scheduler.ADScheduledThreadPoolExecutor$ADScheduledFutureTask.runPeriodic(ADScheduledThreadPoolExecutor.java:206)
        at com.singularity.ee.util.javaspecific.scheduler.ADScheduledThreadPoolExecutor$ADScheduledFutureTask.run(ADScheduledThreadPoolExecutor.java:236)
        at com.singularity.ee.util.javaspecific.scheduler.ADThreadPoolExecutor$Worker.runTask(ADThreadPoolExecutor.java:694)
        at com.singularity.ee.util.javaspecific.scheduler.ADThreadPoolExecutor$Worker.run(ADThreadPoolExecutor.java:726)
        at java.lang.Thread.run(Thread.java:744)

We installed Platform Admin and Event Service before the DB agent. Can anyone help us to relsolve this issue?

Re: No query data in Databasa statistics

Rajesh_Putta — Wed, 26 Apr 2017 05:03:35 GMT

Hi,

Based on the exception it looks like internal events service which is much needed for showing wait state information on DB might not be running. So

could you please follow below steps and let us know if you see any improvement.

1) Stop the events service by running this command:

On Linux

Navigate to <controller-install-dir>/bin
Stop Events Service using
./controller.sh stop-events-service

wait for 1-2 mins

and then start using

controller.sh start-events-service

On Windows

Navigate to <controller-install-dir>/bin

Stop Events Service using
controller.bat stop-events-service

wait for 1-2 mins

and then start using

controller.bat start-events-service

If events service is installed as windows service, with elevated privileges(Run As Administrator)

From services.msc, stop and start the service corresponding to controller events service

Wait for 5 mins and then see if you still see any such messages.

Please find the docs link for the same
https://docs.appdynamics.com/display/PRO42/Start+or+Stop+the+Controller#StartorStoptheController-StartandStoptheEventsService

Post that things should be normal and you should be able to see db agent stats reporting to controllerprovide you have given all permissions to the user whom you configured on DB Collectors.

PS : Any query which is taking less than 1 second will not be shown on Database Query tab.

Thanks

Rajesh

Re: No query data in Databasa statistics

CommunityUser — Thu, 27 Apr 2017 08:06:28 GMT

Hi Rajesh,

It didn't help. Made steps:

[dynamics@app-dyn ~]$ cd /home/dynamics/appdynamics/controller/bin/
[dynamics@app-dyn bin]$ ./controller.sh stop-events-service
[INFO] Path to mysql is not defined in system environment (MYSQL_BIN). Using mysql from /home/dynamics/appdynamics/controller/db/bin

[INFO] Mysql configuration file is not defined in system environment (AD_DB_CNF). Using mysql configuration file: /home/dynamics/appdynamics/controller/db/db.cnf

[INFO] Mysql datadir is /home/dynamics/appdynamics/controller/db/data
[2017-04-27T12:53:27,298+05:00] HV000001: Hibernate Validator 5.2.2.Final
[2017-04-27T12:53:27,450+05:00] [APPLICATION_HOME] has been overridden to [/home/dynamics/appdynamics/controller/events_service/bin/..]
[dynamics@app-dyn bin]$ ./controller.sh stop-events-service
[INFO] Path to mysql is not defined in system environment (MYSQL_BIN). Using mysql from /home/dynamics/appdynamics/controller/db/bin

[INFO] Mysql configuration file is not defined in system environment (AD_DB_CNF). Using mysql configuration file: /home/dynamics/appdynamics/controller/db/db.cnf

[INFO] Mysql datadir is /home/dynamics/appdynamics/controller/db/data
[2017-04-27T12:53:33,931+05:00] HV000001: Hibernate Validator 5.2.2.Final
[2017-04-27T12:53:34,072+05:00] [APPLICATION_HOME] has been overridden to [/home/dynamics/appdynamics/controller/events_service/bin/..]
[dynamics@app-dyn bin]$ date
Thu Apr 27 13:55:08 YEKT 2017
[dynamics@app-dyn bin]$ ./controller.sh start-events-service
[INFO] Path to mysql is not defined in system environment (MYSQL_BIN). Using mysql from /home/dynamics/appdynamics/controller/db/bin

[INFO] Mysql configuration file is not defined in system environment (AD_DB_CNF). Using mysql configuration file: /home/dynamics/appdynamics/controller/db/db.cnf

[INFO] Mysql datadir is /home/dynamics/appdynamics/controller/db/data
Starting the Events Service... This may take a few minutes. Check the /home/dynamics/appdynamics/controller/logs/events-service-startup.log for details.
[dynamics@app-dyn bin]$ tail -50 /home/dynamics/appdynamics/controller/logs/events-service-startup.log

12:55:23.776 [main] INFO c.a.a.processor.AnalyticsService - Starting analytics processor with arguments [-p, /home/dynamics/appdynamics/controller/events_service/conf/events-service-api-store.properties, -y, /home/dynamics/appdynamics/controller/events_service/bin/../conf/events-service-api-store.yml]
namics/controller/events_service/bin/..]
[2017-04-27T12:55:23,545+05:00] Program arguments:

|-Xmx768m|-Xms768m|-Djava.net.preferIPv4Stack=true|-Dfile.encoding=UTF-8|-XX:+UseParNewGC|-XX:+UseConcMarkSweepGC|-XX:CMSInitiatingOccupancyFraction=75|-XX:+UseCMSInitiatingOccupancyOnly|-XX:+DisableExplicitGC|-XX:+PrintGCDetails|-XX:+PrintGCDateStamps|-XX:+PrintGCTimeStamps|-XX:+PrintClassHistogram|-XX:+PrintTenuringDistribution|-XX:+PrintGCApplicationStoppedTime|-XX:+PrintPromotionFailure|-verbose:gc|-XX:GCLogFileSize=64m|-XX:NumberOfGCLogFiles=4|-XX:+UseGCLogFileRotation|-Xloggc:"/home/dynamics/appdynamics/controller/events_service/bin/../logs/events-service-api-store-gc.log"|-DAPPLICATION_HOME="/home/dynamics/appdynamics/controller/events_service/bin/.."|-classpath|"/home/dynamics/appdynamics/controller/events_service/bin/../lib/*"|com.appdynamics.analytics.processor.AnalyticsService|-p|"/home/dynamics/appdynamics/controller/events_service/conf/events-service-api-store.properties"|-y|"/home/dynamics/appdynamics/controller/events_service/bin/../conf/events-service-api-store.yml"
[dynamics@app-dyn bin]$ date
Thu Apr 27 13:55:46 YEKT 2017
[dynamics@app-dyn bin]$ date
Thu Apr 27 13:55:48 YEKT 2017

[dynamics@app-dyn bin]$ tail -50 /home/dynamics/appdynamics/controller/logs/events-service-startup.log

12:55:23.776 [main] INFO c.a.a.processor.AnalyticsService - Starting analytics processor with arguments [-p, /home/dynamics/appdynamics/controller/events_service/conf/events-service-api-store.properties, -y, /home/dynamics/appdynamics/controller/events_service/bin/../conf/events-service-api-store.yml]
namics/controller/events_service/bin/..]
[2017-04-27T12:55:23,545+05:00] Program arguments:

|-Xmx768m|-Xms768m|-Djava.net.preferIPv4Stack=true|-Dfile.encoding=UTF-8|-XX:+UseParNewGC|-XX:+UseConcMarkSweepGC|-XX:CMSInitiatingOccupancyFraction=75|-XX:+UseCMSInitiatingOccupancyOnly|-XX:+DisableExplicitGC|-XX:+PrintGCDetails|-XX:+PrintGCDateStamps|-XX:+PrintGCTimeStamps|-XX:+PrintClassHistogram|-XX:+PrintTenuringDistribution|-XX:+PrintGCApplicationStoppedTime|-XX:+PrintPromotionFailure|-verbose:gc|-XX:GCLogFileSize=64m|-XX:NumberOfGCLogFiles=4|-XX:+UseGCLogFileRotation|-Xloggc:"/home/dynamics/appdynamics/controller/events_service/bin/../logs/events-service-api-store-gc.log"|-DAPPLICATION_HOME="/home/dynamics/appdynamics/controller/events_service/bin/.."|-classpath|"/home/dynamics/appdynamics/controller/events_service/bin/../lib/*"|com.appdynamics.analytics.processor.AnalyticsService|-p|"/home/dynamics/appdynamics/controller/events_service/conf/events-service-api-store.properties"|-y|"/home/dynamics/appdynamics/controller/events_service/bin/../conf/events-service-api-store.yml"

Errors from db-agent log:

[Event-Uploader-Scheduler-1] 27 Apr 2017 13:55:05,687 WARN DBMeasurementUploader - Fail to upload Wait Time Measurements:
com.singularity.ee.rest.ResponseReadException: Error in controller in processing binary request Wait Time Measurement - null
        at com.singularity.ee.rest.controller.request.ABinaryControllerRequest.<init>(ABinaryControllerRequest.java:65)
        at com.singularity.ee.rest.controller.request.dbmon.WaitTimeMeasurementCollectorRequest.<init>(WaitTimeMeasurementCollectorRequest.java:34)
        at com.singularity.ee.agent.dbagent.task.reporter.DBMeasurementUploader.uploadWaitTimeMeasurements(DBMeasurementUploader.java:79)
        at com.singularity.ee.agent.dbagent.task.reporter.DBMeasurementUploader.run(DBMeasurementUploader.java:133)
        at com.singularity.ee.util.javaspecific.scheduler.AgentScheduledExecutorServiceImpl$SafeRunnable.run(AgentScheduledExecutorServiceImpl.java:122)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at com.singularity.ee.util.javaspecific.scheduler.ADFutureTask$Sync.innerRunAndReset(ADFutureTask.java:335)
        at com.singularity.ee.util.javaspecific.scheduler.ADFutureTask.runAndReset(ADFutureTask.java:152)
        at com.singularity.ee.util.javaspecific.scheduler.ADScheduledThreadPoolExecutor$ADScheduledFutureTask.access$101(ADScheduledThreadPoolExecutor.java:119)
        at com.singularity.ee.util.javaspecific.scheduler.ADScheduledThreadPoolExecutor$ADScheduledFutureTask.runPeriodic(ADScheduledThreadPoolExecutor.java:206)
        at com.singularity.ee.util.javaspecific.scheduler.ADScheduledThreadPoolExecutor$ADScheduledFutureTask.run(ADScheduledThreadPoolExecutor.java:236)
        at com.singularity.ee.util.javaspecific.scheduler.ADThreadPoolExecutor$Worker.runTask(ADThreadPoolExecutor.java:694)
        at com.singularity.ee.util.javaspecific.scheduler.ADThreadPoolExecutor$Worker.run(ADThreadPoolExecutor.java:726)
        at java.lang.Thread.run(Thread.java:744)
[Event-Uploader-Scheduler-1] 27 Apr 2017 13:55:05,699 WARN DBMeasurementUploader - Fail to upload Query Stats Measurements:
com.singularity.ee.rest.ResponseReadException: Error in controller in processing binary request Write Query Stats - null
        at com.singularity.ee.rest.controller.request.ABinaryControllerRequest.<init>(ABinaryControllerRequest.java:65)
        at com.singularity.ee.rest.controller.request.dbmon.WriteQueryStatsRequest.<init>(WriteQueryStatsRequest.java:30)
        at com.singularity.ee.agent.dbagent.task.reporter.DBMeasurementUploader.uploadrQueryStats(DBMeasurementUploader.java:119)
        at com.singularity.ee.agent.dbagent.task.reporter.DBMeasurementUploader.run(DBMeasurementUploader.java:137)
        at com.singularity.ee.util.javaspecific.scheduler.AgentScheduledExecutorServiceImpl$SafeRunnable.run(AgentScheduledExecutorServiceImpl.java:122)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at com.singularity.ee.util.javaspecific.scheduler.ADFutureTask$Sync.innerRunAndReset(ADFutureTask.java:335)
        at com.singularity.ee.util.javaspecific.scheduler.ADFutureTask.runAndReset(ADFutureTask.java:152)
        at com.singularity.ee.util.javaspecific.scheduler.ADScheduledThreadPoolExecutor$ADScheduledFutureTask.access$101(ADScheduledThreadPoolExecutor.java:119)
        at com.singularity.ee.util.javaspecific.scheduler.ADScheduledThreadPoolExecutor$ADScheduledFutureTask.runPeriodic(ADScheduledThreadPoolExecutor.java:206)
        at com.singularity.ee.util.javaspecific.scheduler.ADScheduledThreadPoolExecutor$ADScheduledFutureTask.run(ADScheduledThreadPoolExecutor.java:236)
        at com.singularity.ee.util.javaspecific.scheduler.ADThreadPoolExecutor$Worker.runTask(ADThreadPoolExecutor.java:694)
        at com.singularity.ee.util.javaspecific.scheduler.ADThreadPoolExecutor$Worker.run(ADThreadPoolExecutor.java:726)
        at java.lang.Thread.run(Thread.java:744)
[Event-Uploader-Scheduler-1] 27 Apr 2017 13:58:08,361 ERROR DBMeasurementUploader - Fatal transport error while connecting to URL [/controller/instance/7/collectWaitTimes]: java.net.SocketTimeoutException: Read timed out
[Event-Uploader-Scheduler-1] 27 Apr 2017 13:58:08,361 WARN DBMeasurementUploader - Fail to upload Wait Time Measurements: Fatal transport error while connecting to URL [/controller/instance/7/collectWaitTimes]
[Event-Uploader-Scheduler-1] 27 Apr 2017 13:59:08,424 ERROR DBMeasurementUploader - Fatal transport error while connecting to URL [/controller/instance/7/writeQueryStats]: java.net.SocketTimeoutException: Read timed out
[Event-Uploader-Scheduler-1] 27 Apr 2017 13:59:08,425 WARN DBMeasurementUploader - Fail to upload Query Stats Measurements: Fatal transport error while connecting to URL [/controller/instance/7/writeQueryStats]
[Event-Uploader-Scheduler-1] 27 Apr 2017 14:00:18,480 ERROR DBMeasurementUploader - Fatal transport error while connecting to URL [/controller/instance/7/collectWaitTimes]: java.net.SocketTimeoutException: Read timed out
[Event-Uploader-Scheduler-1] 27 Apr 2017 14:00:18,480 WARN DBMeasurementUploader - Fail to upload Wait Time Measurements: Fatal transport error while connecting to URL [/controller/instance/7/collectWaitTimes]
[Event-Uploader-Scheduler-1] 27 Apr 2017 14:01:18,544 ERROR DBMeasurementUploader - Fatal transport error while connecting to URL [/controller/instance/7/writeQueryStats]: java.net.SocketTimeoutException: Read timed out
[Event-Uploader-Scheduler-1] 27 Apr 2017 14:01:18,544 WARN DBMeasurementUploader - Fail to upload Query Stats Measurements: Fatal transport error while connecting to URL [/controller/instance/7/writeQueryStats]
[Event-Uploader-Scheduler-1] 27 Apr 2017 14:02:28,601 ERROR DBMeasurementUploader - Fatal transport error while connecting to URL [/controller/instance/7/collectWaitTimes]: java.net.SocketTimeoutException: Read timed out
[Event-Uploader-Scheduler-1] 27 Apr 2017 14:02:28,601 WARN DBMeasurementUploader - Fail to upload Wait Time Measurements: Fatal transport error while connecting to URL [/controller/instance/7/collectWaitTimes]

We installed platform admin and events service on this server with separated MySQL db.

Re: No query data in Databasa statistics

Rajesh_Putta — Mon, 01 May 2017 11:53:12 GMT

Hi,

Thanks for the update. Could you please send us details as listed here

1) Zipped logs folder located under /home/dynamics/appdynamics/controller

2) DB Agent logs folder

Thanks

Rajesh

Re: No query data in Databasa statistics

CommunityUser — Tue, 02 May 2017 08:35:38 GMT

Hi Rajesh,

The logs are attached to the post. Delete some server.log_... files (with repeated messages) as logs file is to big for a post.

Re: No query data in Databasa statistics

Mohammed_Rayan — Wed, 03 May 2017 05:32:14 GMT

Hi,

Can you share the output of the below URL's. You can run this on the same host where your events-service is up and running

curl http://<events_service_machine>:9081/healthcheck?pretty=true
curl http://<events_service_machine>:9200/_cat/shards?v
curl http://<events_service_machine>:9200/_cat/indices?v

Note: 9081 is the admin port and 9200 is cluster port of the events-service

Regards,

Mohammed Rayan

Re: No query data in Databasa statistics

CommunityUser — Wed, 03 May 2017 10:05:28 GMT

Hi Rajesh,

The results are:

[dynamics@app-dyn ~]$ curl http://localhost:9081/healthcheck?pretty=true
{
"deadlocks" : {
    "healthy" : true
},
"events-service-api-store / BizOutcome [stage.extract]" : {
    "healthy" : true,
    "message" : "Rates (Avg per second. Avg of last 5 min) accepted events: [0.000000], discarded events: [0.000000], acceptance ratio: [0.000000]"
},
"events-service-api-store / BizOutcome [stage.filter]" : {
    "healthy" : true,
    "message" : "Rates (Avg per second. Avg of last 5 min) accepted events: [0.000000], discarded events: [0.000000], acceptance ratio: [0.000000]"
},
"events-service-api-store / Build information" : {
    "healthy" : true,
    "message" : "buildName=123-4.3.0.next-build, buildHash=d0e4fc0167eff6b5fa974d1f9a8180e5a1e89044, buildTimestamp=2017-03-17T22:06:29+0000, buildNumber=3709, jobName=build-events-service, version=4.3.0.1"
},
"events-service-api-store / Chunking.AllChunksTimer [CatchAllChunkingStrategy]" : {
    "healthy" : true,
    "message" : "Duration in seconds percentile distribution, 50%: [0.000000], 75%: [0.000000], 95%: [0.000000], 99%: [0.000000], mean: [0.000000], count: [0.000000]"
},
"events-service-api-store / Chunking.AllChunksTimer [SeriesChunkingStrategy]" : {
    "healthy" : true,
    "message" : "Duration in seconds percentile distribution, 50%: [0.000000], 75%: [0.000000], 95%: [0.000000], 99%: [0.000000], mean: [0.000000], count: [0.000000]"
},
"events-service-api-store / Chunking.AllChunksTimer [TimeRangeChunkingStrategy]" : {
    "healthy" : true,
    "message" : "Duration in seconds percentile distribution, 50%: [0.000000], 75%: [0.000000], 95%: [0.000000], 99%: [0.000000], mean: [0.000000], count: [0.000000]"
},
"events-service-api-store / Chunking.Counts [CatchAllChunkingStrategy]" : {
    "healthy" : true,
    "message" : "Rates (Avg per second. Avg of last 5 min) success: [0.000000], user error: [0.000000], timeout: [0.000000], error: [0.000000]"
},
"events-service-api-store / Chunking.Counts [SeriesChunkingStrategy]" : {
    "healthy" : true,
    "message" : "Rates (Avg per second. Avg of last 5 min) success: [0.000000], user error: [0.000000], timeout: [0.000000], error: [0.000000]"
},
"events-service-api-store / Chunking.Counts [TimeRangeChunkingStrategy]" : {
    "healthy" : true,
    "message" : "Rates (Avg per second. Avg of last 5 min) success: [0.000000], user error: [0.000000], timeout: [0.000000], error: [0.000000]"
},
"events-service-api-store / Chunking.FirstChunkTimer [CatchAllChunkingStrategy]" : {
    "healthy" : true,
    "message" : "Duration in seconds percentile distribution, 50%: [0.000000], 75%: [0.000000], 95%: [0.000000], 99%: [0.000000], mean: [0.000000], count: [0.000000]"
},
"events-service-api-store / Chunking.FirstChunkTimer [SeriesChunkingStrategy]" : {
    "healthy" : true,
    "message" : "Duration in seconds percentile distribution, 50%: [0.000000], 75%: [0.000000], 95%: [0.000000], 99%: [0.000000], mean: [0.000000], count: [0.000000]"
},
"events-service-api-store / Chunking.FirstChunkTimer [TimeRangeChunkingStrategy]" : {
    "healthy" : true,
    "message" : "Duration in seconds percentile distribution, 50%: [0.000000], 75%: [0.000000], 95%: [0.000000], 99%: [0.000000], mean: [0.000000], count: [0.000000]"
},
"events-service-api-store / Chunking.LastChunkTimer [CatchAllChunkingStrategy]" : {
    "healthy" : true,
    "message" : "Duration in seconds percentile distribution, 50%: [0.000000], 75%: [0.000000], 95%: [0.000000], 99%: [0.000000], mean: [0.000000], count: [0.000000]"
},
"events-service-api-store / Chunking.LastChunkTimer [SeriesChunkingStrategy]" : {
    "healthy" : true,
    "message" : "Duration in seconds percentile distribution, 50%: [0.000000], 75%: [0.000000], 95%: [0.000000], 99%: [0.000000], mean: [0.000000], count: [0.000000]"
},

Re: No query data in Databasa statistics

CommunityUser — Wed, 03 May 2017 10:06:04 GMT

"events-service-api-store / Chunking.LastChunkTimer [TimeRangeChunkingStrategy]" : {
    "healthy" : true,
    "message" : "Duration in seconds percentile distribution, 50%: [0.000000], 75%: [0.000000], 95%: [0.000000], 99%: [0.000000], mean: [0.000000], count: [0.000000]"
},
"events-service-api-store / Chunking.TotalChunksTimer [CatchAllChunkingStrategy]" : {
    "healthy" : true,
    "message" : "Duration in seconds percentile distribution, 50%: [0.000000], 75%: [0.000000], 95%: [0.000000], 99%: [0.000000], mean: [0.000000], count: [0.000000]"
},
"events-service-api-store / Chunking.TotalChunksTimer [SeriesChunkingStrategy]" : {
    "healthy" : true,
    "message" : "Duration in seconds percentile distribution, 50%: [0.000000], 75%: [0.000000], 95%: [0.000000], 99%: [0.000000], mean: [0.000000], count: [0.000000]"
},
"events-service-api-store / Chunking.TotalChunksTimer [TimeRangeChunkingStrategy]" : {
    "healthy" : true,
    "message" : "Duration in seconds percentile distribution, 50%: [0.000000], 75%: [0.000000], 95%: [0.000000], 99%: [0.000000], mean: [0.000000], count: [0.000000]"
},
"events-service-api-store / Configuration properties" : {
    "healthy" : true,
    "message" : "Dynamic properties: [query.chunking.strategies.time.range.splits=[86400000, 604800000, 2592000000], schema.validator.max.custom.events.per.account=20, query.chunking.num.worker.threads=[10, 20, 30, 40], query.chunking.enabled=true, query.chunking.strategies.series.limits=[10, 20, 30, 40], query.chunking.query.timeout.ms=600000, query.chunking.strategies.catch.all.limits=[86400000, 604800000, 2592000000], query.chunking.more.logging=true, query.chunking.async.dump.interval.ms=0, query.chunking.min.time.ms=43200000, publish.validator.max.fields.per.event=255, query.validator.max.batch.count=20, query.chunking.strategies=[SeriesChunkingStrategy, TimeRangeChunkingStrategy], query.chunking.strategies.progress.heuristic.ms=604800000]"
},
"events-service-api-store / Index compaction management" : {
    "healthy" : true,
    "message" : "Healthcheck for [Index compaction management]: Is running flag is: [false], last execution successful flag is: [true]"
},
"events-service-api-store / Job Scan management" : {
    "healthy" : true,
    "message" : "Healthcheck for [Job Scan management]: Is running flag is: [false], last execution successful flag is: [true]"
},
"events-service-api-store / Resource [entitysearch/query]" : {
    "healthy" : true,
    "message" : "Rates (Avg per second. Avg of last 5 min) success: [0.000000], user error: [0.000000], timeout: [0.000000], error: [0.000000]"
},
"events-service-api-store / Resource [entitysearch/sync]" : {
    "healthy" : true,
    "message" : "Rates (Avg per second. Avg of last 5 min) success: [0.000000], user error: [0.000000], timeout: [0.000000], error: [0.000000]"
},
"events-service-api-store / Resource [events/publish - meter]" : {
    "healthy" : true,
    "message" : "Rates (Avg per second. Avg of last 5 min) success: [0.000000], user error: [0.000000], timeout: [0.000000], error: [0.000000]"
},
"events-service-api-store / Resource [events/publish]" : {
    "healthy" : true,
    "message" : "Rates (Avg per second. Avg of last 5 min) success: [0.000000], user error: [0.000000], timeout: [0.000000], error: [0.000000]"
},
"events-service-api-store / Resource [events/query - meter]" : {
    "healthy" : true,
    "message" : "Rates (Avg per second. Avg of last 5 min) success: [0.000000], user error: [0.000000], timeout: [0.000000], error: [0.000000]"
},
"events-service-api-store / Resource [events/query]" : {
    "healthy" : true,
    "message" : "Rates (Avg per second. Avg of last 5 min) success: [0.000000], user error: [0.000000], timeout: [0.000000], error: [0.000000]"
},
"events-service-api-store / Resource [events/schema - create]" : {
    "healthy" : true,
    "message" : "Rates (Avg per second. Avg of last 5 min) success: [0.000000], user error: [0.000000], timeout: [0.000000], error: [0.000000]"
},
"events-service-api-store / Resource [events/schema - delete]" : {
    "healthy" : true,
    "message" : "Rates (Avg per second. Avg of last 5 min) success: [0.000000], user error: [0.000000], timeout: [0.000000], error: [0.000000]"
},
"events-service-api-store / Resource [events/schema - get]" : {
    "healthy" : true,
    "message" : "Rates (Avg per second. Avg of last 5 min) success: [0.000000], user error: [0.000000], timeout: [0.000000], error: [0.000000]"
},
"events-service-api-store / Resource [events/schema - update]" : {
    "healthy" : true,
    "message" : "Rates (Avg per second. Avg of last 5 min) success: [0.000000], user error: [0.000000], timeout: [0.000000], error: [0.000000]"
},

Re: No query data in Databasa statistics

CommunityUser — Wed, 03 May 2017 10:08:26 GMT

Hi Mohammed,

The result are attached.

Re: No query data in Databasa statistics

Mohammed_Rayan — Wed, 03 May 2017 10:27:49 GMT

Hi there,

Thanks for the output and it's very clear now about why we are seeing the below messages repeatedly about (all shards failed) in your server logs.

Log Snippet:

#|2017-05-02T12:18:05.795+0500|SEVERE|glassfish 4.1|com.singularity.ee.controller.beans.ExceptionHandlingInterceptor|_ThreadID=307;_ThreadName=http-listener-1(5);_TimeMillis=1493709485795;_LevelValue=1000;|Encountered runtime exception
RestException(statusCode=503, code=Service.Unavailable, errorMessage=Service unavailable: [all shards failed], developerMessage=Exception Message Chain : Failed to execute phase [query_fetch], all shards failed
Caused/related: all shards failed

From the healthcheck output,I infer your cluster state is in RED state due to unassigned shards and hence the above error which says "all shards failed" as a result of it.

 "events-service-api-store / elasticsearch-singlenode-module" : {
    "healthy" : false,
    "message" : "Current [appdynamics-events-service-cluster] cluster state: [RED], data nodes: [1], nodes: [1], active shards: [2], relocating shards: [0], initializing shards: [0], unassigned shards: [18], timed out: [false]"
  },

Below Shards output shows which indices are in UNASSIGNED state and all the dbmon event type indices are in UNASSIGNED

[dynamics@app-dyn ~]$ curl http://localhost:9200/_cat/shards?v
index                                                                                               shard prirep state      docs  store ip           node       
async_executor_status                                                                               0     p      UNASSIGNED                                     
appdynamics_version                                                                                 0     p      UNASSIGNED                                     
biz_outcome_definitions                                                                             0     p      UNASSIGNED                                     
event_type_extracted_fields                                                                         0     p      UNASSIGNED                                     
appdynamics_meters_v2                                                                               0     p      UNASSIGNED                                     
event_type_metadata_hidden_fields_v1                                                                0     p      UNASSIGNED                                     
appdynamics_entity_metadata                                                                         0     p      STARTED       0   159b  node-2c7781cf-58e5-4784-b11c-5fc6272af932
customer1_39714636-eec0-4331-932d-1394d05fabd3___dbmon-query-stat___2017-04-27_12-56-07             1     p      UNASSIGNED                                     
customer1_39714636-eec0-4331-932d-1394d05fabd3___dbmon-query-stat___2017-04-27_12-56-07             2     p      UNASSIGNED                                     
customer1_39714636-eec0-4331-932d-1394d05fabd3___dbmon-query-stat___2017-04-27_12-56-07             0     p      UNASSIGNED                                     
customer1_39714636-eec0-4331-932d-1394d05fabd3___dbmon-wait-time___2017-04-27_12-56-05              1     p      UNASSIGNED                                     
customer1_39714636-eec0-4331-932d-1394d05fabd3___dbmon-wait-time___2017-04-27_12-56-05              2     p      UNASSIGNED                                     
customer1_39714636-eec0-4331-932d-1394d05fabd3___dbmon-wait-time___2017-04-27_12-56-05              0     p      UNASSIGNED                                     
customer1_39714636-eec0-4331-932d-1394d05fabd3___dbmon-execution-plan-details___2017-04-27_12-56-08 1     p      UNASSIGNED                                     
customer1_39714636-eec0-4331-932d-1394d05fabd3___dbmon-execution-plan-details___2017-04-27_12-56-08 2     p      UNASSIGNED                                     
customer1_39714636-eec0-4331-932d-1394d05fabd3___dbmon-execution-plan-details___2017-04-27_12-56-08 0     p      UNASSIGNED                                     
slm_performance_configs_v1                                                                          0     p      UNASSIGNED                                     
appdynamics_accounts                                                                                0     p      STARTED       2 17.6kb  node-2c7781cf-58e5-4784-b11c-5fc6272af932
event_type_metadata                                                                                 0     p      UNASSIGNED                                     
appdynamics_api_keys_v1                                                                             0     p      UNASSIGNED

This can happen for a number of reasons and hence to understand the reason behind it, Please execute the following command and share its output

curl -s -XGET http://localhost:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason | grep UNASSIGNED

Regards,

Mohammed Rayan

Re: No query data in Databasa statistics

CommunityUser — Thu, 04 May 2017 08:53:55 GMT

Mohammed,

The result of the command:

[dynamics@app-dyn ~]$ curl -s -XGET http://localhost:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason | grep UNASSIGNED
async_executor_status                                                                               0 p UNASSIGNED NODE_LEFT
appdynamics_version                                                                                 0 p UNASSIGNED NODE_LEFT
biz_outcome_definitions                                                                             0 p UNASSIGNED NODE_LEFT
event_type_extracted_fields                                                                         0 p UNASSIGNED NODE_LEFT
appdynamics_meters_v2                                                                               0 p UNASSIGNED NODE_LEFT
event_type_metadata_hidden_fields_v1                                                                0 p UNASSIGNED NODE_LEFT
customer1_39714636-eec0-4331-932d-1394d05fabd3___dbmon-query-stat___2017-04-27_12-56-07             1 p UNASSIGNED NODE_LEFT
customer1_39714636-eec0-4331-932d-1394d05fabd3___dbmon-query-stat___2017-04-27_12-56-07             2 p UNASSIGNED NODE_LEFT
customer1_39714636-eec0-4331-932d-1394d05fabd3___dbmon-query-stat___2017-04-27_12-56-07             0 p UNASSIGNED NODE_LEFT
customer1_39714636-eec0-4331-932d-1394d05fabd3___dbmon-wait-time___2017-04-27_12-56-05              1 p UNASSIGNED NODE_LEFT
customer1_39714636-eec0-4331-932d-1394d05fabd3___dbmon-wait-time___2017-04-27_12-56-05              2 p UNASSIGNED NODE_LEFT
customer1_39714636-eec0-4331-932d-1394d05fabd3___dbmon-wait-time___2017-04-27_12-56-05              0 p UNASSIGNED NODE_LEFT
customer1_39714636-eec0-4331-932d-1394d05fabd3___dbmon-execution-plan-details___2017-04-27_12-56-08 1 p UNASSIGNED NODE_LEFT
customer1_39714636-eec0-4331-932d-1394d05fabd3___dbmon-execution-plan-details___2017-04-27_12-56-08 2 p UNASSIGNED NODE_LEFT
customer1_39714636-eec0-4331-932d-1394d05fabd3___dbmon-execution-plan-details___2017-04-27_12-56-08 0 p UNASSIGNED NODE_LEFT
slm_performance_configs_v1                                                                          0 p UNASSIGNED NODE_LEFT
event_type_metadata                                                                                 0 p UNASSIGNED NODE_LEFT
appdynamics_api_keys_v1                                                                             0 p UNASSIGNED NODE_LEFT
[dynamics@app-dyn ~]$

Re: No query data in Databasa statistics

Mohammed_Rayan — Thu, 04 May 2017 09:40:45 GMT

Thanks for the update and the reason seems to be node left and this again can be due to many reasons and so can you try to run the below command which will manually allocate the unassigned shards

Run the below command on your events-service host

curl -XPUT 'localhost:9200/_cluster/settings' -d
'{ "transient":
  { "cluster.routing.allocation.enable" : "all" 
  }
}'

Then verify the status using the following commands again and share the output.

curl http://<events_service_machine>:9081/healthcheck?pretty=true

curl http://<events_service_machine>:9200/_cat/shards?v

curl -s -XGET http://localhost:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason,unassigned.details | grep UNASSIGNED

Regards,

Mohammed Rayan

Re: No query data in Databasa statistics

CommunityUser — Thu, 04 May 2017 14:41:31 GMT

Mohammed,

The results are attached

Re: No query data in Databasa statistics

Mohammed_Rayan — Thu, 04 May 2017 15:06:33 GMT

Hi there,

I still see the state of your cluster in RED and shards are still Unassigned.

Have you done a restart after the allocation command of your events-service and if not, can you please re-run the allocation command and restart your events-service and then check the health status using the earlier provided commands

Regards,

Mohammed Rayan

Re: No query data in Databasa statistics

CommunityUser — Wed, 10 May 2017 07:32:43 GMT

Hi Mohammed,

I cannot make the steps because our trial license is expired...