Our current SOAR servers, fresh install on AWS EC2s, 500's each night. Upon investigation, it looks like there's this error in the logs:
File "/opt/soar/usr/python39/lib/python3.9/site-packages/psycopg2/__init__.py", line 127, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: connection to server on socket "/tmp//.s.PGSQL.6432" failed: No such file or directory
Is the server running locally and accepting connections on that socket?
On a healthy server, that file is present. On a 500-error server, it's missing. Is there an explanation of why that might be going missing? Issue is temporarily resolved by stopping and starting phantom again.
I think it might be related to PostgreSQL or pgbouncer.
Same OS? SeLinux turned on or some other company agent on there?? These are the usual culprits for this kind of fun errors 😄
I disabled selinux, fapolicyd, and firewalld, but it still happens. Although, I think we may have narrowed it down to an in-house script that runs nightly! Thanks for the help!
@catherinelam I have not seen this before but it does look Postgres-ey.
Is this a single instance or Hot/Warm standby? If so are you sure the postgres stream is allowed (5432) between them and you have confirmed the sync is working?
The files are definitely Postgres files but I am not sure what action creates them and why they would be deleted during runtime to then "go missing".
I hope you have also raised a support case for this too?
This is a warm standby, and the primary and warm standby show the same behaviour.
Additionally, we have some standalone servers that also show it, so I don't think it's specific to a certain architecture.
I tried opening a support case ticket, but whenever I submit a ticket I just get a blank page and it doesn't go through 😞 I've reached out to a company contact to see if I can escalate the issue.
Thanks for looking!