Server 500 Error: Missing /tmp//.s.PGSQL.6432 File

[catherinelam]

Our current SOAR servers, fresh install on AWS EC2s, 500's each night. Upon investigation, it looks like there's this error in the logs:

File "/opt/soar/usr/python39/lib/python3.9/site-packages/psycopg2/__init__.py", line 127, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: connection to server on socket "/tmp//.s.PGSQL.6432" failed: No such file or directory
Is the server running locally and accepting connections on that socket?

On a healthy server, that file is present. On a 500-error server, it's missing. Is there an explanation of why that might be going missing? Issue is temporarily resolved by stopping and starting phantom again. 

I think it might be related to PostgreSQL or pgbouncer. 

[phanTom]

Same OS? SeLinux turned on or some other company agent on there?? These are the usual culprits for this kind of fun errors 😄 

[catherinelam]

I disabled selinux, fapolicyd, and firewalld, but it still happens. Although, I think we may have narrowed it down to an in-house script that runs nightly! Thanks for the help! 

[phanTom]

@catherinelam I have not seen this before but it does look Postgres-ey. 

Is this a single instance or Hot/Warm standby? If so are you sure the postgres stream is allowed (5432) between them and you have confirmed the sync is working?

The files are definitely Postgres files but I am not sure what action creates them and why they would be deleted during runtime to then "go missing". 

I hope you have also raised a support case for this too?

[catherinelam]

This is a warm standby, and the primary and warm standby show the same behaviour. 

Additionally, we have some standalone servers that also show it, so I don't think it's specific to a certain architecture. 

I tried opening a support case ticket, but whenever I submit a ticket I just get a blank page and it doesn't go through 😞 I've reached out to a company contact to see if I can escalate the issue. 

Thanks for looking!