Splunk SOAR

Server 500 Error: Missing /tmp//.s.PGSQL.6432 File

catherinelam
Explorer

Our current SOAR servers, fresh install on AWS EC2s, 500's each night. Upon investigation, it looks like there's this error in the logs:

File "/opt/soar/usr/python39/lib/python3.9/site-packages/psycopg2/__init__.py", line 127, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: connection to server on socket "/tmp//.s.PGSQL.6432" failed: No such file or directory
Is the server running locally and accepting connections on that socket?

On a healthy server, that file is present. On a 500-error server, it's missing. Is there an explanation of why that might be going missing? Issue is temporarily resolved by stopping and starting phantom again. 

I think it might be related to PostgreSQL or pgbouncer. 

Labels (1)
Tags (3)
0 Karma

phanTom
SplunkTrust
SplunkTrust

Same OS? SeLinux turned on or some other company agent on there?? These are the usual culprits for this kind of fun errors 😄 

0 Karma

catherinelam
Explorer

I disabled selinux, fapolicyd, and firewalld, but it still happens. Although, I think we may have narrowed it down to an in-house script that runs nightly! Thanks for the help! 

0 Karma

phanTom
SplunkTrust
SplunkTrust

@catherinelam I have not seen this before but it does look Postgres-ey. 

Is this a single instance or Hot/Warm standby? If so are you sure the postgres stream is allowed (5432) between them and you have confirmed the sync is working?

The files are definitely Postgres files but I am not sure what action creates them and why they would be deleted during runtime to then "go missing". 

I hope you have also raised a support case for this too?

 

0 Karma

catherinelam
Explorer

This is a warm standby, and the primary and warm standby show the same behaviour. 

Additionally, we have some standalone servers that also show it, so I don't think it's specific to a certain architecture. 

I tried opening a support case ticket, but whenever I submit a ticket I just get a blank page and it doesn't go through 😞 I've reached out to a company contact to see if I can escalate the issue. 

Thanks for looking!

0 Karma
Get Updates on the Splunk Community!

How to Monitor Google Kubernetes Engine (GKE)

We’ve looked at how to integrate Kubernetes environments with Splunk Observability Cloud, but what about ...

Index This | How can you make 45 using only 4?

October 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with this ...

Splunk Education Goes to Washington | Splunk GovSummit 2024

If you’re in the Washington, D.C. area, this is your opportunity to take your career and Splunk skills to the ...