Hi,
I can't access to the Controller http://<url>:8090/controller/.
In the server.log:
[#|2015-05-20T07:14:15.001+0200|WARNING|glassfish3.1.2|com.appdynamics.METRICS.WRITE|_ThreadID=132;_ThreadName=Thread-5;|Controller is low on disk space! Disk space left: 1007 MB|#]
[#|2015-05-20T07:14:15.248+0200|SEVERE|glassfish3.1.2|com.appdynamics.METRICS.WRITE|_ThreadID=132;_ThreadName=Thread-5;|Controller shutting down due to very low disk space! Space left is 1007MB|#]
I saw that the file system is filled to 93% and the directories that occupy the most space are:
1] /production/AppDynamics/Controller/events_service/analytics-processor/data/appdynamics-analytics-cluster/nodes/0/indices/active_static_index___2015-02-05_14-39-33/...
2] /production/AppDynamics/Controller/db/data/controller
What can I delete to free up space?
How can I do it?
Regards,
Teresa
Hi Teresa,
We see this is due to internal feature of AD Controller to shutdown glassfish appserver when memory available is less , though we have an option to disable the memory available check we do not recommend it as that leads to controller db table corruption.
We would like to check how it goes with below suggestions, check if that makes any difference:
'- delete any logs under (leave server.log, database.log delete old files such as server.log_ as such)
- delete any unnecessary files outside controller directory
- do "not" delete any files under events_service as that will result in loosing of data
- See how it goes after above changes, we might try truncating some tables manually but that is the last option here, like snapshot as such, but check deleting old logs helps here
Regards,
Arun
Under /production/AppDynamics/Controller/logs I don't delete server.log but only the old files server.log_*.
Now I have this log:
-rw-rw-r-- 1 apm apm 0 Feb 5 12:04 analytics-all-http.log
-rw-rw-r-- 1 apm apm 400K Mar 28 00:00 analytics-all.log1497570497530968.tmp
-rw-rw-r-- 1 apm apm 8.0K Apr 1 00:00 analytics-all.log1839588173558360.tmp
-rw-rw-r-- 1 apm apm 8.0K Apr 2 00:00 analytics-all.log1925989047025406.tmp
-rw-rw-r-- 1 apm apm 0 Apr 2 23:59 analytics-all.log2012377954609728.tmp
-rw-rw-r-- 1 apm apm 4.0K Apr 6 00:00 analytics-all.log2271592343261299.tmp
-rw-rw-r-- 1 apm apm 0 Apr 7 00:00 analytics-all.log2357993168929141.tmp
-rw-rw-r-- 1 apm apm 3.7K May 12 16:16 reporting-server.log
-rw-r----- 1 apm apm 41K May 19 17:22 database.log
-rw-rw-r-- 1 apm apm 938 May 19 17:40 startAS.log
-rw-rw-r-- 1 apm apm 2.6M May 20 07:14 server.log
-rw-rw-r-- 1 apm apm 37K May 21 00:00 analytics-all-2015-05-20.log.gz
-rw-rw-r-- 1 apm apm 35K May 22 00:00 analytics-all-2015-05-21.log.gz
-rw-rw-r-- 1 apm apm 35K May 23 00:00 analytics-all-2015-05-22.log.gz
-rw-rw-r-- 1 apm apm 35K May 24 00:00 analytics-all-2015-05-23.log.gz
-rw-rw-r-- 1 apm apm 34K May 25 00:00 analytics-all-2015-05-24.log.gz
-rw-rw-r-- 1 apm apm 17M May 25 15:22 analytics-all-gc.log.0
-rw-rw-r-- 1 apm apm 750K May 25 15:26 analytics-all.log
but the size of the file system is 92%.
You say "delete any unnecessary files outside controller directory" but the directory that occupy the most space is /production/AppDynamics/Controller:
[apm@dsetlab120 AppDynamics]$ pwd
/production/AppDynamics
[apm@dsetlab120 AppDynamics]$ du -h --max-depth=1
211M ./EUEM
25G ./Controller
153M ./dbagent
26G .
I find this documentation https://docs.appdynamics.com/display/PRO14S/Database+Size+and+Data+Retention where:
"You can change the amount of data you retain in the Controller database by changing the retention period for events, snapshots, and metric data. If you notice the Controller database growing too large, consider lowering your retention settings. You can also purge old data that is no longer needed by lowering the retention settings."
Do you recommend to change the retention period?
Regards,
Teresa
Hi Teresa,
Earlier i have thought that the controller appserver is not running for sometime say 20 minutes and going down or not able to start hence i have not pointed , you are right on the retention settings given that controller is going down after sometime, request you to optimize the values for metrics.retention.period, snapshots, and events related retention settings and the contorller will purge data at contorller db level and this changes will take 20-30 mts and if controller runs fine for this time, you should see some space released, Hope that clarifies.