AppD Archive

Controller shutting down

CommunityUser
Splunk Employee
Splunk Employee

Hi,
I can't access to the Controller http://<url>:8090/controller/.

In the server.log:

[#|2015-05-20T07:14:15.001+0200|WARNING|glassfish3.1.2|com.appdynamics.METRICS.WRITE|_ThreadID=132;_ThreadName=Thread-5;|Controller is low on disk space! Disk space left: 1007 MB|#]

[#|2015-05-20T07:14:15.248+0200|SEVERE|glassfish3.1.2|com.appdynamics.METRICS.WRITE|_ThreadID=132;_ThreadName=Thread-5;|Controller shutting down due to very low disk space! Space left is 1007MB|#]

I saw that the file system is filled to 93% and the directories that occupy the most space are:

1] /production/AppDynamics/Controller/events_service/analytics-processor/data/appdynamics-analytics-cluster/nodes/0/indices/active_static_index___2015-02-05_14-39-33/...
2] /production/AppDynamics/Controller/db/data/controller

What can I delete to free up space?
How can I do it?

Regards,

Teresa

0 Karma

Arun_Dasetty
Super Champion

Hi Teresa,

 

We see this is due to internal feature of AD Controller to shutdown glassfish appserver when memory available is less , though we have an option to disable the memory available check we do not recommend it as that leads to controller db table corruption.

 

We would like to check how it goes with below suggestions, check if that makes any difference:

'- delete any logs under (leave server.log, database.log delete old files such as server.log_ as such) 

- delete any unnecessary files outside controller directory 

- do "not" delete any files under events_service as that will result in loosing of data

- See how it goes after above changes, we might try truncating some tables manually but that is the last option here, like snapshot as such, but check deleting old logs helps here

 

Regards,

Arun

0 Karma

CommunityUser
Splunk Employee
Splunk Employee

Under /production/AppDynamics/Controller/logs I don't delete server.log but only the old files server.log_*.

Now I have this log:
-rw-rw-r-- 1 apm apm 0 Feb 5 12:04 analytics-all-http.log
-rw-rw-r-- 1 apm apm 400K Mar 28 00:00 analytics-all.log1497570497530968.tmp
-rw-rw-r-- 1 apm apm 8.0K Apr 1 00:00 analytics-all.log1839588173558360.tmp
-rw-rw-r-- 1 apm apm 8.0K Apr 2 00:00 analytics-all.log1925989047025406.tmp
-rw-rw-r-- 1 apm apm 0 Apr 2 23:59 analytics-all.log2012377954609728.tmp
-rw-rw-r-- 1 apm apm 4.0K Apr 6 00:00 analytics-all.log2271592343261299.tmp
-rw-rw-r-- 1 apm apm 0 Apr 7 00:00 analytics-all.log2357993168929141.tmp
-rw-rw-r-- 1 apm apm 3.7K May 12 16:16 reporting-server.log
-rw-r----- 1 apm apm 41K May 19 17:22 database.log
-rw-rw-r-- 1 apm apm 938 May 19 17:40 startAS.log
-rw-rw-r-- 1 apm apm 2.6M May 20 07:14 server.log
-rw-rw-r-- 1 apm apm 37K May 21 00:00 analytics-all-2015-05-20.log.gz
-rw-rw-r-- 1 apm apm 35K May 22 00:00 analytics-all-2015-05-21.log.gz
-rw-rw-r-- 1 apm apm 35K May 23 00:00 analytics-all-2015-05-22.log.gz
-rw-rw-r-- 1 apm apm 35K May 24 00:00 analytics-all-2015-05-23.log.gz
-rw-rw-r-- 1 apm apm 34K May 25 00:00 analytics-all-2015-05-24.log.gz
-rw-rw-r-- 1 apm apm 17M May 25 15:22 analytics-all-gc.log.0
-rw-rw-r-- 1 apm apm 750K May 25 15:26 analytics-all.log

but the size of the file system is 92%.

You say "delete any unnecessary files outside controller directory" but the directory that occupy the most space is /production/AppDynamics/Controller:


[apm@dsetlab120 AppDynamics]$ pwd
/production/AppDynamics

[apm@dsetlab120 AppDynamics]$ du -h --max-depth=1
211M ./EUEM
25G ./Controller
153M ./dbagent
26G .


I find this documentation https://docs.appdynamics.com/display/PRO14S/Database+Size+and+Data+Retention where:
"You can change the amount of data you retain in the Controller database by changing the retention period for events, snapshots, and metric data. If you notice the Controller database growing too large, consider lowering your retention settings. You can also purge old data that is no longer needed by lowering the retention settings."

Do you recommend to change the retention period?

Regards,
Teresa

0 Karma

Arun_Dasetty
Super Champion

Hi Teresa,

 

Earlier i have thought that the controller appserver is not running for sometime say 20 minutes and going down or not able to start hence i have not pointed , you are right on the retention settings given that controller is going down after sometime, request you to optimize the values for metrics.retention.period, snapshots, and events related retention settings and the contorller will purge data at contorller db level and this changes will take 20-30 mts and if controller runs fine for this time, you should see some space released, Hope that clarifies.

0 Karma
Get Updates on the Splunk Community!

See just what you’ve been missing | Observability tracks at Splunk University

Looking to sharpen your observability skills so you can better understand how to collect and analyze data from ...

Weezer at .conf25? Say it ain’t so!

Hello Splunkers, The countdown to .conf25 is on-and we've just turned up the volume! We're thrilled to ...

How SC4S Makes Suricata Logs Ingestion Simple

Network security monitoring has become increasingly critical for organizations of all sizes. Splunk has ...