No query data in Databasa statistics

CommunityUser · ‎04-25-2017

Hello!

We are making POC of AppDynamics with our Customer and we've met the following issue. The dbagent shows Wait Time graph in AppDynamics GUI but shows no SQL queries and "Internal Error 500" when we try to open Queries tab. In agent.log we see the following error:

25 Apr 2017 14:40:05,756 WARN [Event-Uploader-Scheduler-2] DBMeasurementUploader:81 - Fail to upload Wait Time Measurements:
com.singularity.ee.rest.ResponseReadException: Error in controller in processing binary request Wait Time Measurement - null
        at com.singularity.ee.rest.controller.request.ABinaryControllerRequest.<init>(ABinaryControllerRequest.java:65)
        at com.singularity.ee.rest.controller.request.dbmon.WaitTimeMeasurementCollectorRequest.<init>(WaitTimeMeasurementCollectorRequest.java:34)
        at com.singularity.ee.agent.dbagent.task.reporter.DBMeasurementUploader.uploadWaitTimeMeasurements(DBMeasurementUploader.java:79)
        at com.singularity.ee.agent.dbagent.task.reporter.DBMeasurementUploader.run(DBMeasurementUploader.java:133)
        at com.singularity.ee.util.javaspecific.scheduler.AgentScheduledExecutorServiceImpl$SafeRunnable.run(AgentScheduledExecutorServiceImpl.java:122)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at com.singularity.ee.util.javaspecific.scheduler.ADFutureTask$Sync.innerRunAndReset(ADFutureTask.java:335)
        at com.singularity.ee.util.javaspecific.scheduler.ADFutureTask.runAndReset(ADFutureTask.java:152)
        at com.singularity.ee.util.javaspecific.scheduler.ADScheduledThreadPoolExecutor$ADScheduledFutureTask.access$101(ADScheduledThreadPoolExecutor.java:119)
        at com.singularity.ee.util.javaspecific.scheduler.ADScheduledThreadPoolExecutor$ADScheduledFutureTask.runPeriodic(ADScheduledThreadPoolExecutor.java:206)
        at com.singularity.ee.util.javaspecific.scheduler.ADScheduledThreadPoolExecutor$ADScheduledFutureTask.run(ADScheduledThreadPoolExecutor.java:236)
        at com.singularity.ee.util.javaspecific.scheduler.ADThreadPoolExecutor$Worker.runTask(ADThreadPoolExecutor.java:694)
        at com.singularity.ee.util.javaspecific.scheduler.ADThreadPoolExecutor$Worker.run(ADThreadPoolExecutor.java:726)
        at java.lang.Thread.run(Thread.java:744)

We installed Platform Admin and Event Service before the DB agent. Can anyone help us to relsolve this issue?

CommunityUser · ‎05-10-2017

Hi Mohammed,

I cannot make the steps because our trial license is expired...

CommunityUser · ‎05-04-2017

Mohammed,

The results are attached

Mohammed_Rayan · ‎05-04-2017

Hi there,

I still see the state of your cluster in RED and shards are still Unassigned.

Have you done a restart after the allocation command of your events-service and if not, can you please re-run the allocation command and restart your events-service and then check the health status using the earlier provided commands

Regards,

Mohammed Rayan

CommunityUser · ‎05-04-2017

Mohammed,

The result of the command:

[dynamics@app-dyn ~]$ curl -s -XGET http://localhost:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason | grep UNASSIGNED
async_executor_status                                                                               0 p UNASSIGNED NODE_LEFT
appdynamics_version                                                                                 0 p UNASSIGNED NODE_LEFT
biz_outcome_definitions                                                                             0 p UNASSIGNED NODE_LEFT
event_type_extracted_fields                                                                         0 p UNASSIGNED NODE_LEFT
appdynamics_meters_v2                                                                               0 p UNASSIGNED NODE_LEFT
event_type_metadata_hidden_fields_v1                                                                0 p UNASSIGNED NODE_LEFT
customer1_39714636-eec0-4331-932d-1394d05fabd3___dbmon-query-stat___2017-04-27_12-56-07             1 p UNASSIGNED NODE_LEFT
customer1_39714636-eec0-4331-932d-1394d05fabd3___dbmon-query-stat___2017-04-27_12-56-07             2 p UNASSIGNED NODE_LEFT
customer1_39714636-eec0-4331-932d-1394d05fabd3___dbmon-query-stat___2017-04-27_12-56-07             0 p UNASSIGNED NODE_LEFT
customer1_39714636-eec0-4331-932d-1394d05fabd3___dbmon-wait-time___2017-04-27_12-56-05              1 p UNASSIGNED NODE_LEFT
customer1_39714636-eec0-4331-932d-1394d05fabd3___dbmon-wait-time___2017-04-27_12-56-05              2 p UNASSIGNED NODE_LEFT
customer1_39714636-eec0-4331-932d-1394d05fabd3___dbmon-wait-time___2017-04-27_12-56-05              0 p UNASSIGNED NODE_LEFT
customer1_39714636-eec0-4331-932d-1394d05fabd3___dbmon-execution-plan-details___2017-04-27_12-56-08 1 p UNASSIGNED NODE_LEFT
customer1_39714636-eec0-4331-932d-1394d05fabd3___dbmon-execution-plan-details___2017-04-27_12-56-08 2 p UNASSIGNED NODE_LEFT
customer1_39714636-eec0-4331-932d-1394d05fabd3___dbmon-execution-plan-details___2017-04-27_12-56-08 0 p UNASSIGNED NODE_LEFT
slm_performance_configs_v1                                                                          0 p UNASSIGNED NODE_LEFT
event_type_metadata                                                                                 0 p UNASSIGNED NODE_LEFT
appdynamics_api_keys_v1                                                                             0 p UNASSIGNED NODE_LEFT
[dynamics@app-dyn ~]$

Mohammed_Rayan · ‎05-04-2017

Thanks for the update and the reason seems to be node left and this again can be due to many reasons and so can you try to run the below command which will manually allocate the unassigned shards

Run the below command on your events-service host

curl -XPUT 'localhost:9200/_cluster/settings' -d
'{ "transient":
  { "cluster.routing.allocation.enable" : "all" 
  }
}'

Then verify the status using the following commands again and share the output.

curl http://<events_service_machine>:9081/healthcheck?pretty=true

curl http://<events_service_machine>:9200/_cat/shards?v

curl -s -XGET http://localhost:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason,unassigned.details | grep UNASSIGNED

Regards,

Mohammed Rayan

CommunityUser · ‎05-03-2017

Hi Mohammed,

The result are attached.

Mohammed_Rayan · ‎05-03-2017

Hi there,

Thanks for the output and it's very clear now about why we are seeing the below messages repeatedly about (all shards failed) in your server logs.

Log Snippet:

#|2017-05-02T12:18:05.795+0500|SEVERE|glassfish 4.1|com.singularity.ee.controller.beans.ExceptionHandlingInterceptor|_ThreadID=307;_ThreadName=http-listener-1(5);_TimeMillis=1493709485795;_LevelValue=1000;|Encountered runtime exception
RestException(statusCode=503, code=Service.Unavailable, errorMessage=Service unavailable: [all shards failed], developerMessage=Exception Message Chain : Failed to execute phase [query_fetch], all shards failed
Caused/related: all shards failed

From the healthcheck output,I infer your cluster state is in RED state due to unassigned shards and hence the above error which says "all shards failed" as a result of it.

 "events-service-api-store / elasticsearch-singlenode-module" : {
    "healthy" : false,
    "message" : "Current [appdynamics-events-service-cluster] cluster state: [RED], data nodes: [1], nodes: [1], active shards: [2], relocating shards: [0], initializing shards: [0], unassigned shards: [18], timed out: [false]"
  },

Below Shards output shows which indices are in UNASSIGNED state and all the dbmon event type indices are in UNASSIGNED

[dynamics@app-dyn ~]$ curl http://localhost:9200/_cat/shards?v
index                                                                                               shard prirep state      docs  store ip           node       
async_executor_status                                                                               0     p      UNASSIGNED                                     
appdynamics_version                                                                                 0     p      UNASSIGNED                                     
biz_outcome_definitions                                                                             0     p      UNASSIGNED                                     
event_type_extracted_fields                                                                         0     p      UNASSIGNED                                     
appdynamics_meters_v2                                                                               0     p      UNASSIGNED                                     
event_type_metadata_hidden_fields_v1                                                                0     p      UNASSIGNED                                     
appdynamics_entity_metadata                                                                         0     p      STARTED       0   159b  node-2c7781cf-58e5-4784-b11c-5fc6272af932
customer1_39714636-eec0-4331-932d-1394d05fabd3___dbmon-query-stat___2017-04-27_12-56-07             1     p      UNASSIGNED                                     
customer1_39714636-eec0-4331-932d-1394d05fabd3___dbmon-query-stat___2017-04-27_12-56-07             2     p      UNASSIGNED                                     
customer1_39714636-eec0-4331-932d-1394d05fabd3___dbmon-query-stat___2017-04-27_12-56-07             0     p      UNASSIGNED                                     
customer1_39714636-eec0-4331-932d-1394d05fabd3___dbmon-wait-time___2017-04-27_12-56-05              1     p      UNASSIGNED                                     
customer1_39714636-eec0-4331-932d-1394d05fabd3___dbmon-wait-time___2017-04-27_12-56-05              2     p      UNASSIGNED                                     
customer1_39714636-eec0-4331-932d-1394d05fabd3___dbmon-wait-time___2017-04-27_12-56-05              0     p      UNASSIGNED                                     
customer1_39714636-eec0-4331-932d-1394d05fabd3___dbmon-execution-plan-details___2017-04-27_12-56-08 1     p      UNASSIGNED                                     
customer1_39714636-eec0-4331-932d-1394d05fabd3___dbmon-execution-plan-details___2017-04-27_12-56-08 2     p      UNASSIGNED                                     
customer1_39714636-eec0-4331-932d-1394d05fabd3___dbmon-execution-plan-details___2017-04-27_12-56-08 0     p      UNASSIGNED                                     
slm_performance_configs_v1                                                                          0     p      UNASSIGNED                                     
appdynamics_accounts                                                                                0     p      STARTED       2 17.6kb  node-2c7781cf-58e5-4784-b11c-5fc6272af932
event_type_metadata                                                                                 0     p      UNASSIGNED                                     
appdynamics_api_keys_v1                                                                             0     p      UNASSIGNED

This can happen for a number of reasons and hence to understand the reason behind it, Please execute the following command and share its output

curl -s -XGET http://localhost:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason | grep UNASSIGNED

Regards,

Mohammed Rayan

CommunityUser · ‎05-03-2017

"events-service-api-store / Chunking.LastChunkTimer [TimeRangeChunkingStrategy]" : {
    "healthy" : true,
    "message" : "Duration in seconds percentile distribution, 50%: [0.000000], 75%: [0.000000], 95%: [0.000000], 99%: [0.000000], mean: [0.000000], count: [0.000000]"
},
"events-service-api-store / Chunking.TotalChunksTimer [CatchAllChunkingStrategy]" : {
    "healthy" : true,
    "message" : "Duration in seconds percentile distribution, 50%: [0.000000], 75%: [0.000000], 95%: [0.000000], 99%: [0.000000], mean: [0.000000], count: [0.000000]"
},
"events-service-api-store / Chunking.TotalChunksTimer [SeriesChunkingStrategy]" : {
    "healthy" : true,
    "message" : "Duration in seconds percentile distribution, 50%: [0.000000], 75%: [0.000000], 95%: [0.000000], 99%: [0.000000], mean: [0.000000], count: [0.000000]"
},
"events-service-api-store / Chunking.TotalChunksTimer [TimeRangeChunkingStrategy]" : {
    "healthy" : true,
    "message" : "Duration in seconds percentile distribution, 50%: [0.000000], 75%: [0.000000], 95%: [0.000000], 99%: [0.000000], mean: [0.000000], count: [0.000000]"
},
"events-service-api-store / Configuration properties" : {
    "healthy" : true,
    "message" : "Dynamic properties: [query.chunking.strategies.time.range.splits=[86400000, 604800000, 2592000000], schema.validator.max.custom.events.per.account=20, query.chunking.num.worker.threads=[10, 20, 30, 40], query.chunking.enabled=true, query.chunking.strategies.series.limits=[10, 20, 30, 40], query.chunking.query.timeout.ms=600000, query.chunking.strategies.catch.all.limits=[86400000, 604800000, 2592000000], query.chunking.more.logging=true, query.chunking.async.dump.interval.ms=0, query.chunking.min.time.ms=43200000, publish.validator.max.fields.per.event=255, query.validator.max.batch.count=20, query.chunking.strategies=[SeriesChunkingStrategy, TimeRangeChunkingStrategy], query.chunking.strategies.progress.heuristic.ms=604800000]"
},
"events-service-api-store / Index compaction management" : {
    "healthy" : true,
    "message" : "Healthcheck for [Index compaction management]: Is running flag is: [false], last execution successful flag is: [true]"
},
"events-service-api-store / Job Scan management" : {
    "healthy" : true,
    "message" : "Healthcheck for [Job Scan management]: Is running flag is: [false], last execution successful flag is: [true]"
},
"events-service-api-store / Resource [entitysearch/query]" : {
    "healthy" : true,
    "message" : "Rates (Avg per second. Avg of last 5 min) success: [0.000000], user error: [0.000000], timeout: [0.000000], error: [0.000000]"
},
"events-service-api-store / Resource [entitysearch/sync]" : {
    "healthy" : true,
    "message" : "Rates (Avg per second. Avg of last 5 min) success: [0.000000], user error: [0.000000], timeout: [0.000000], error: [0.000000]"
},
"events-service-api-store / Resource [events/publish - meter]" : {
    "healthy" : true,
    "message" : "Rates (Avg per second. Avg of last 5 min) success: [0.000000], user error: [0.000000], timeout: [0.000000], error: [0.000000]"
},
"events-service-api-store / Resource [events/publish]" : {
    "healthy" : true,
    "message" : "Rates (Avg per second. Avg of last 5 min) success: [0.000000], user error: [0.000000], timeout: [0.000000], error: [0.000000]"
},
"events-service-api-store / Resource [events/query - meter]" : {
    "healthy" : true,
    "message" : "Rates (Avg per second. Avg of last 5 min) success: [0.000000], user error: [0.000000], timeout: [0.000000], error: [0.000000]"
},
"events-service-api-store / Resource [events/query]" : {
    "healthy" : true,
    "message" : "Rates (Avg per second. Avg of last 5 min) success: [0.000000], user error: [0.000000], timeout: [0.000000], error: [0.000000]"
},
"events-service-api-store / Resource [events/schema - create]" : {
    "healthy" : true,
    "message" : "Rates (Avg per second. Avg of last 5 min) success: [0.000000], user error: [0.000000], timeout: [0.000000], error: [0.000000]"
},
"events-service-api-store / Resource [events/schema - delete]" : {
    "healthy" : true,
    "message" : "Rates (Avg per second. Avg of last 5 min) success: [0.000000], user error: [0.000000], timeout: [0.000000], error: [0.000000]"
},
"events-service-api-store / Resource [events/schema - get]" : {
    "healthy" : true,
    "message" : "Rates (Avg per second. Avg of last 5 min) success: [0.000000], user error: [0.000000], timeout: [0.000000], error: [0.000000]"
},
"events-service-api-store / Resource [events/schema - update]" : {
    "healthy" : true,
    "message" : "Rates (Avg per second. Avg of last 5 min) success: [0.000000], user error: [0.000000], timeout: [0.000000], error: [0.000000]"
},

CommunityUser · ‎05-03-2017

Hi Rajesh,

The results are:

[dynamics@app-dyn ~]$ curl http://localhost:9081/healthcheck?pretty=true
{
"deadlocks" : {
    "healthy" : true
},
"events-service-api-store / BizOutcome [stage.extract]" : {
    "healthy" : true,
    "message" : "Rates (Avg per second. Avg of last 5 min) accepted events: [0.000000], discarded events: [0.000000], acceptance ratio: [0.000000]"
},
"events-service-api-store / BizOutcome [stage.filter]" : {
    "healthy" : true,
    "message" : "Rates (Avg per second. Avg of last 5 min) accepted events: [0.000000], discarded events: [0.000000], acceptance ratio: [0.000000]"
},
"events-service-api-store / Build information" : {
    "healthy" : true,
    "message" : "buildName=123-4.3.0.next-build, buildHash=d0e4fc0167eff6b5fa974d1f9a8180e5a1e89044, buildTimestamp=2017-03-17T22:06:29+0000, buildNumber=3709, jobName=build-events-service, version=4.3.0.1"
},
"events-service-api-store / Chunking.AllChunksTimer [CatchAllChunkingStrategy]" : {
    "healthy" : true,
    "message" : "Duration in seconds percentile distribution, 50%: [0.000000], 75%: [0.000000], 95%: [0.000000], 99%: [0.000000], mean: [0.000000], count: [0.000000]"
},
"events-service-api-store / Chunking.AllChunksTimer [SeriesChunkingStrategy]" : {
    "healthy" : true,
    "message" : "Duration in seconds percentile distribution, 50%: [0.000000], 75%: [0.000000], 95%: [0.000000], 99%: [0.000000], mean: [0.000000], count: [0.000000]"
},
"events-service-api-store / Chunking.AllChunksTimer [TimeRangeChunkingStrategy]" : {
    "healthy" : true,
    "message" : "Duration in seconds percentile distribution, 50%: [0.000000], 75%: [0.000000], 95%: [0.000000], 99%: [0.000000], mean: [0.000000], count: [0.000000]"
},
"events-service-api-store / Chunking.Counts [CatchAllChunkingStrategy]" : {
    "healthy" : true,
    "message" : "Rates (Avg per second. Avg of last 5 min) success: [0.000000], user error: [0.000000], timeout: [0.000000], error: [0.000000]"
},
"events-service-api-store / Chunking.Counts [SeriesChunkingStrategy]" : {
    "healthy" : true,
    "message" : "Rates (Avg per second. Avg of last 5 min) success: [0.000000], user error: [0.000000], timeout: [0.000000], error: [0.000000]"
},
"events-service-api-store / Chunking.Counts [TimeRangeChunkingStrategy]" : {
    "healthy" : true,
    "message" : "Rates (Avg per second. Avg of last 5 min) success: [0.000000], user error: [0.000000], timeout: [0.000000], error: [0.000000]"
},
"events-service-api-store / Chunking.FirstChunkTimer [CatchAllChunkingStrategy]" : {
    "healthy" : true,
    "message" : "Duration in seconds percentile distribution, 50%: [0.000000], 75%: [0.000000], 95%: [0.000000], 99%: [0.000000], mean: [0.000000], count: [0.000000]"
},
"events-service-api-store / Chunking.FirstChunkTimer [SeriesChunkingStrategy]" : {
    "healthy" : true,
    "message" : "Duration in seconds percentile distribution, 50%: [0.000000], 75%: [0.000000], 95%: [0.000000], 99%: [0.000000], mean: [0.000000], count: [0.000000]"
},
"events-service-api-store / Chunking.FirstChunkTimer [TimeRangeChunkingStrategy]" : {
    "healthy" : true,
    "message" : "Duration in seconds percentile distribution, 50%: [0.000000], 75%: [0.000000], 95%: [0.000000], 99%: [0.000000], mean: [0.000000], count: [0.000000]"
},
"events-service-api-store / Chunking.LastChunkTimer [CatchAllChunkingStrategy]" : {
    "healthy" : true,
    "message" : "Duration in seconds percentile distribution, 50%: [0.000000], 75%: [0.000000], 95%: [0.000000], 99%: [0.000000], mean: [0.000000], count: [0.000000]"
},
"events-service-api-store / Chunking.LastChunkTimer [SeriesChunkingStrategy]" : {
    "healthy" : true,
    "message" : "Duration in seconds percentile distribution, 50%: [0.000000], 75%: [0.000000], 95%: [0.000000], 99%: [0.000000], mean: [0.000000], count: [0.000000]"
},

CommunityUser · ‎05-02-2017

Hi Rajesh,

The logs are attached to the post. Delete some server.log_... files (with repeated messages) as logs file is to big for a post.

Mohammed_Rayan · ‎05-02-2017

Hi,

Can you share the output of the below URL's. You can run this on the same host where your events-service is up and running

curl http://<events_service_machine>:9081/healthcheck?pretty=true
curl http://<events_service_machine>:9200/_cat/shards?v
curl http://<events_service_machine>:9200/_cat/indices?v

Note: 9081 is the admin port and 9200 is cluster port of the events-service

Regards,

Mohammed Rayan

CommunityUser · ‎04-27-2017

Hi Rajesh,

It didn't help. Made steps:

[dynamics@app-dyn ~]$ cd /home/dynamics/appdynamics/controller/bin/
[dynamics@app-dyn bin]$ ./controller.sh stop-events-service
[INFO] Path to mysql is not defined in system environment (MYSQL_BIN). Using mysql from /home/dynamics/appdynamics/controller/db/bin

[INFO] Mysql configuration file is not defined in system environment (AD_DB_CNF). Using mysql configuration file: /home/dynamics/appdynamics/controller/db/db.cnf

[INFO] Mysql datadir is /home/dynamics/appdynamics/controller/db/data
[2017-04-27T12:53:27,298+05:00] HV000001: Hibernate Validator 5.2.2.Final
[2017-04-27T12:53:27,450+05:00] [APPLICATION_HOME] has been overridden to [/home/dynamics/appdynamics/controller/events_service/bin/..]
[dynamics@app-dyn bin]$ ./controller.sh stop-events-service
[INFO] Path to mysql is not defined in system environment (MYSQL_BIN). Using mysql from /home/dynamics/appdynamics/controller/db/bin

[INFO] Mysql configuration file is not defined in system environment (AD_DB_CNF). Using mysql configuration file: /home/dynamics/appdynamics/controller/db/db.cnf

[INFO] Mysql datadir is /home/dynamics/appdynamics/controller/db/data
[2017-04-27T12:53:33,931+05:00] HV000001: Hibernate Validator 5.2.2.Final
[2017-04-27T12:53:34,072+05:00] [APPLICATION_HOME] has been overridden to [/home/dynamics/appdynamics/controller/events_service/bin/..]
[dynamics@app-dyn bin]$ date
Thu Apr 27 13:55:08 YEKT 2017
[dynamics@app-dyn bin]$ ./controller.sh start-events-service
[INFO] Path to mysql is not defined in system environment (MYSQL_BIN). Using mysql from /home/dynamics/appdynamics/controller/db/bin

[INFO] Mysql configuration file is not defined in system environment (AD_DB_CNF). Using mysql configuration file: /home/dynamics/appdynamics/controller/db/db.cnf

[INFO] Mysql datadir is /home/dynamics/appdynamics/controller/db/data
Starting the Events Service... This may take a few minutes. Check the /home/dynamics/appdynamics/controller/logs/events-service-startup.log for details.
[dynamics@app-dyn bin]$ tail -50 /home/dynamics/appdynamics/controller/logs/events-service-startup.log

12:55:23.776 [main] INFO c.a.a.processor.AnalyticsService - Starting analytics processor with arguments [-p, /home/dynamics/appdynamics/controller/events_service/conf/events-service-api-store.properties, -y, /home/dynamics/appdynamics/controller/events_service/bin/../conf/events-service-api-store.yml]
namics/controller/events_service/bin/..]
[2017-04-27T12:55:23,545+05:00] Program arguments:

|-Xmx768m|-Xms768m|-Djava.net.preferIPv4Stack=true|-Dfile.encoding=UTF-8|-XX:+UseParNewGC|-XX:+UseConcMarkSweepGC|-XX:CMSInitiatingOccupancyFraction=75|-XX:+UseCMSInitiatingOccupancyOnly|-XX:+DisableExplicitGC|-XX:+PrintGCDetails|-XX:+PrintGCDateStamps|-XX:+PrintGCTimeStamps|-XX:+PrintClassHistogram|-XX:+PrintTenuringDistribution|-XX:+PrintGCApplicationStoppedTime|-XX:+PrintPromotionFailure|-verbose:gc|-XX:GCLogFileSize=64m|-XX:NumberOfGCLogFiles=4|-XX:+UseGCLogFileRotation|-Xloggc:"/home/dynamics/appdynamics/controller/events_service/bin/../logs/events-service-api-store-gc.log"|-DAPPLICATION_HOME="/home/dynamics/appdynamics/controller/events_service/bin/.."|-classpath|"/home/dynamics/appdynamics/controller/events_service/bin/../lib/*"|com.appdynamics.analytics.processor.AnalyticsService|-p|"/home/dynamics/appdynamics/controller/events_service/conf/events-service-api-store.properties"|-y|"/home/dynamics/appdynamics/controller/events_service/bin/../conf/events-service-api-store.yml"
[dynamics@app-dyn bin]$ date
Thu Apr 27 13:55:46 YEKT 2017
[dynamics@app-dyn bin]$ date
Thu Apr 27 13:55:48 YEKT 2017

[dynamics@app-dyn bin]$ tail -50 /home/dynamics/appdynamics/controller/logs/events-service-startup.log

12:55:23.776 [main] INFO c.a.a.processor.AnalyticsService - Starting analytics processor with arguments [-p, /home/dynamics/appdynamics/controller/events_service/conf/events-service-api-store.properties, -y, /home/dynamics/appdynamics/controller/events_service/bin/../conf/events-service-api-store.yml]
namics/controller/events_service/bin/..]
[2017-04-27T12:55:23,545+05:00] Program arguments:

|-Xmx768m|-Xms768m|-Djava.net.preferIPv4Stack=true|-Dfile.encoding=UTF-8|-XX:+UseParNewGC|-XX:+UseConcMarkSweepGC|-XX:CMSInitiatingOccupancyFraction=75|-XX:+UseCMSInitiatingOccupancyOnly|-XX:+DisableExplicitGC|-XX:+PrintGCDetails|-XX:+PrintGCDateStamps|-XX:+PrintGCTimeStamps|-XX:+PrintClassHistogram|-XX:+PrintTenuringDistribution|-XX:+PrintGCApplicationStoppedTime|-XX:+PrintPromotionFailure|-verbose:gc|-XX:GCLogFileSize=64m|-XX:NumberOfGCLogFiles=4|-XX:+UseGCLogFileRotation|-Xloggc:"/home/dynamics/appdynamics/controller/events_service/bin/../logs/events-service-api-store-gc.log"|-DAPPLICATION_HOME="/home/dynamics/appdynamics/controller/events_service/bin/.."|-classpath|"/home/dynamics/appdynamics/controller/events_service/bin/../lib/*"|com.appdynamics.analytics.processor.AnalyticsService|-p|"/home/dynamics/appdynamics/controller/events_service/conf/events-service-api-store.properties"|-y|"/home/dynamics/appdynamics/controller/events_service/bin/../conf/events-service-api-store.yml"

Errors from db-agent log:

[Event-Uploader-Scheduler-1] 27 Apr 2017 13:55:05,687 WARN DBMeasurementUploader - Fail to upload Wait Time Measurements:
com.singularity.ee.rest.ResponseReadException: Error in controller in processing binary request Wait Time Measurement - null
        at com.singularity.ee.rest.controller.request.ABinaryControllerRequest.<init>(ABinaryControllerRequest.java:65)
        at com.singularity.ee.rest.controller.request.dbmon.WaitTimeMeasurementCollectorRequest.<init>(WaitTimeMeasurementCollectorRequest.java:34)
        at com.singularity.ee.agent.dbagent.task.reporter.DBMeasurementUploader.uploadWaitTimeMeasurements(DBMeasurementUploader.java:79)
        at com.singularity.ee.agent.dbagent.task.reporter.DBMeasurementUploader.run(DBMeasurementUploader.java:133)
        at com.singularity.ee.util.javaspecific.scheduler.AgentScheduledExecutorServiceImpl$SafeRunnable.run(AgentScheduledExecutorServiceImpl.java:122)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at com.singularity.ee.util.javaspecific.scheduler.ADFutureTask$Sync.innerRunAndReset(ADFutureTask.java:335)
        at com.singularity.ee.util.javaspecific.scheduler.ADFutureTask.runAndReset(ADFutureTask.java:152)
        at com.singularity.ee.util.javaspecific.scheduler.ADScheduledThreadPoolExecutor$ADScheduledFutureTask.access$101(ADScheduledThreadPoolExecutor.java:119)
        at com.singularity.ee.util.javaspecific.scheduler.ADScheduledThreadPoolExecutor$ADScheduledFutureTask.runPeriodic(ADScheduledThreadPoolExecutor.java:206)
        at com.singularity.ee.util.javaspecific.scheduler.ADScheduledThreadPoolExecutor$ADScheduledFutureTask.run(ADScheduledThreadPoolExecutor.java:236)
        at com.singularity.ee.util.javaspecific.scheduler.ADThreadPoolExecutor$Worker.runTask(ADThreadPoolExecutor.java:694)
        at com.singularity.ee.util.javaspecific.scheduler.ADThreadPoolExecutor$Worker.run(ADThreadPoolExecutor.java:726)
        at java.lang.Thread.run(Thread.java:744)
[Event-Uploader-Scheduler-1] 27 Apr 2017 13:55:05,699 WARN DBMeasurementUploader - Fail to upload Query Stats Measurements:
com.singularity.ee.rest.ResponseReadException: Error in controller in processing binary request Write Query Stats - null
        at com.singularity.ee.rest.controller.request.ABinaryControllerRequest.<init>(ABinaryControllerRequest.java:65)
        at com.singularity.ee.rest.controller.request.dbmon.WriteQueryStatsRequest.<init>(WriteQueryStatsRequest.java:30)
        at com.singularity.ee.agent.dbagent.task.reporter.DBMeasurementUploader.uploadrQueryStats(DBMeasurementUploader.java:119)
        at com.singularity.ee.agent.dbagent.task.reporter.DBMeasurementUploader.run(DBMeasurementUploader.java:137)
        at com.singularity.ee.util.javaspecific.scheduler.AgentScheduledExecutorServiceImpl$SafeRunnable.run(AgentScheduledExecutorServiceImpl.java:122)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at com.singularity.ee.util.javaspecific.scheduler.ADFutureTask$Sync.innerRunAndReset(ADFutureTask.java:335)
        at com.singularity.ee.util.javaspecific.scheduler.ADFutureTask.runAndReset(ADFutureTask.java:152)
        at com.singularity.ee.util.javaspecific.scheduler.ADScheduledThreadPoolExecutor$ADScheduledFutureTask.access$101(ADScheduledThreadPoolExecutor.java:119)
        at com.singularity.ee.util.javaspecific.scheduler.ADScheduledThreadPoolExecutor$ADScheduledFutureTask.runPeriodic(ADScheduledThreadPoolExecutor.java:206)
        at com.singularity.ee.util.javaspecific.scheduler.ADScheduledThreadPoolExecutor$ADScheduledFutureTask.run(ADScheduledThreadPoolExecutor.java:236)
        at com.singularity.ee.util.javaspecific.scheduler.ADThreadPoolExecutor$Worker.runTask(ADThreadPoolExecutor.java:694)
        at com.singularity.ee.util.javaspecific.scheduler.ADThreadPoolExecutor$Worker.run(ADThreadPoolExecutor.java:726)
        at java.lang.Thread.run(Thread.java:744)
[Event-Uploader-Scheduler-1] 27 Apr 2017 13:58:08,361 ERROR DBMeasurementUploader - Fatal transport error while connecting to URL [/controller/instance/7/collectWaitTimes]: java.net.SocketTimeoutException: Read timed out
[Event-Uploader-Scheduler-1] 27 Apr 2017 13:58:08,361 WARN DBMeasurementUploader - Fail to upload Wait Time Measurements: Fatal transport error while connecting to URL [/controller/instance/7/collectWaitTimes]
[Event-Uploader-Scheduler-1] 27 Apr 2017 13:59:08,424 ERROR DBMeasurementUploader - Fatal transport error while connecting to URL [/controller/instance/7/writeQueryStats]: java.net.SocketTimeoutException: Read timed out
[Event-Uploader-Scheduler-1] 27 Apr 2017 13:59:08,425 WARN DBMeasurementUploader - Fail to upload Query Stats Measurements: Fatal transport error while connecting to URL [/controller/instance/7/writeQueryStats]
[Event-Uploader-Scheduler-1] 27 Apr 2017 14:00:18,480 ERROR DBMeasurementUploader - Fatal transport error while connecting to URL [/controller/instance/7/collectWaitTimes]: java.net.SocketTimeoutException: Read timed out
[Event-Uploader-Scheduler-1] 27 Apr 2017 14:00:18,480 WARN DBMeasurementUploader - Fail to upload Wait Time Measurements: Fatal transport error while connecting to URL [/controller/instance/7/collectWaitTimes]
[Event-Uploader-Scheduler-1] 27 Apr 2017 14:01:18,544 ERROR DBMeasurementUploader - Fatal transport error while connecting to URL [/controller/instance/7/writeQueryStats]: java.net.SocketTimeoutException: Read timed out
[Event-Uploader-Scheduler-1] 27 Apr 2017 14:01:18,544 WARN DBMeasurementUploader - Fail to upload Query Stats Measurements: Fatal transport error while connecting to URL [/controller/instance/7/writeQueryStats]
[Event-Uploader-Scheduler-1] 27 Apr 2017 14:02:28,601 ERROR DBMeasurementUploader - Fatal transport error while connecting to URL [/controller/instance/7/collectWaitTimes]: java.net.SocketTimeoutException: Read timed out
[Event-Uploader-Scheduler-1] 27 Apr 2017 14:02:28,601 WARN DBMeasurementUploader - Fail to upload Wait Time Measurements: Fatal transport error while connecting to URL [/controller/instance/7/collectWaitTimes]

We installed platform admin and events service on this server with separated MySQL db.

Rajesh_Putta · ‎05-01-2017

Hi,

Thanks for the update. Could you please send us details as listed here

1) Zipped logs folder located under /home/dynamics/appdynamics/controller

2) DB Agent logs folder

Thanks

Rajesh

Rajesh_Putta · ‎04-25-2017

Hi,

Based on the exception it looks like internal events service which is much needed for showing wait state information on DB might not be running. So

could you please follow below steps and let us know if you see any improvement.

1) Stop the events service by running this command:

On Linux

Navigate to <controller-install-dir>/bin
Stop Events Service using
./controller.sh stop-events-service

wait for 1-2 mins

and then start using

controller.sh start-events-service

On Windows

Navigate to <controller-install-dir>/bin

Stop Events Service using
controller.bat stop-events-service

wait for 1-2 mins

and then start using

controller.bat start-events-service

If events service is installed as windows service, with elevated privileges(Run As Administrator)

From services.msc, stop and start the service corresponding to controller events service

Wait for 5 mins and then see if you still see any such messages.

Please find the docs link for the same
https://docs.appdynamics.com/display/PRO42/Start+or+Stop+the+Controller#StartorStoptheController-Sta...

Post that things should be normal and you should be able to see db agent stats reporting to controllerprovide you have given all permissions to the user whom you configured on DB Collectors.

PS : Any query which is taking less than 1 second will not be shown on Database Query tab.

Thanks

Rajesh