Phantom System Health Check

[rsantoso_splunk]

Would like to ask on how can we determine if the System Health being shown is still within threshold and will not affect the Phantom performance.

How can we determine if we have good or bad thresholds in our System Health and if we need to do changes in our server set-up or troubleshoot running processes?

Eg. If Memory Usage of the system and the processes are still OK.

[rsantoso_splunk · Accepted solution]

The system health here is basically the reflection of the phantom instance OS Memory, Load average and Disk Usage. Also the utilisation of the memory and CPU of the processes. There is no threshold where it is impacting phantom performance as long as the resource still available. The utilisation can be more than 90% and Phantom still working fine because the resources are still there.

Having said that there is Linux kernel Out of Memory (OOM) on keeping the score of each process. This is more from the Operating System Linux. Thus if a certain process reach the score threshold, this process will be killed. Please refer to the following url for information on OOM:
https://dev.to/rrampage/surviving-the-linux-oom-killer-2ki9

Phantom is an automated response platform. If it is operating at 70-80% of system capacity, it is to be expected that an event surge will buckle the VM. This is not best practice. Ideally, headroom for surge events should be planned for, and VM specs adjusted accordingly.

Additionally, There is script that can be use to monitor the container status threshold using REST API:
https://my.phantom.us/kb/47/
This would require user login. You would need to register to phantom on getting the login credentials.