topic Re: Server 500 Error: Missing /tmp//.s.PGSQL.6432 File in Splunk SOAR

Server 500 Error: Missing /tmp//.s.PGSQL.6432 File

catherinelam — Tue, 30 Jan 2024 16:47:10 GMT

Our current SOAR servers, fresh install on AWS EC2s, 500's each night. Upon investigation, it looks like there's this error in the logs:

File "/opt/soar/usr/python39/lib/python3.9/site-packages/psycopg2/__init__.py", line 127, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: connection to server on socket "/tmp//.s.PGSQL.6432" failed: No such file or directory
Is the server running locally and accepting connections on that socket?

On a healthy server, that file is present. On a 500-error server, it's missing. Is there an explanation of why that might be going missing? Issue is temporarily resolved by stopping and starting phantom again.

I think it might be related to PostgreSQL or pgbouncer.

Re: Server 500 Error: Missing /tmp//.s.PGSQL.6432 File

phanTom — Tue, 30 Jan 2024 17:08:27 GMT

@catherinelam I have not seen this before but it does look Postgres-ey.

Is this a single instance or Hot/Warm standby? If so are you sure the postgres stream is allowed (5432) between them and you have confirmed the sync is working?

The files are definitely Postgres files but I am not sure what action creates them and why they would be deleted during runtime to then "go missing".

I hope you have also raised a support case for this too?

Re: Server 500 Error: Missing /tmp//.s.PGSQL.6432 File

catherinelam — Tue, 30 Jan 2024 17:12:16 GMT

This is a warm standby, and the primary and warm standby show the same behaviour.

Additionally, we have some standalone servers that also show it, so I don't think it's specific to a certain architecture.

I tried opening a support case ticket, but whenever I submit a ticket I just get a blank page and it doesn't go through 😞 I've reached out to a company contact to see if I can escalate the issue.

Thanks for looking!

Re: Server 500 Error: Missing /tmp//.s.PGSQL.6432 File

phanTom — Tue, 30 Jan 2024 17:15:26 GMT

Same OS? SeLinux turned on or some other company agent on there?? These are the usual culprits for this kind of fun errors 😄

Re: Server 500 Error: Missing /tmp//.s.PGSQL.6432 File

catherinelam — Tue, 06 Feb 2024 21:21:14 GMT

I disabled selinux, fapolicyd, and firewalld, but it still happens. Although, I think we may have narrowed it down to an in-house script that runs nightly! Thanks for the help!