Splunk SOAR (f.k.a. Phantom)

Server 500 Error: Missing /tmp//.s.PGSQL.6432 File

catherinelam
Loves-to-Learn

Our current SOAR servers, fresh install on AWS EC2s, 500's each night. Upon investigation, it looks like there's this error in the logs:

File "/opt/soar/usr/python39/lib/python3.9/site-packages/psycopg2/__init__.py", line 127, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: connection to server on socket "/tmp//.s.PGSQL.6432" failed: No such file or directory
Is the server running locally and accepting connections on that socket?

On a healthy server, that file is present. On a 500-error server, it's missing. Is there an explanation of why that might be going missing? Issue is temporarily resolved by stopping and starting phantom again. 

I think it might be related to PostgreSQL or pgbouncer. 

Labels (1)
Tags (3)
0 Karma

phanTom
SplunkTrust
SplunkTrust

Same OS? SeLinux turned on or some other company agent on there?? These are the usual culprits for this kind of fun errors 😄 

0 Karma

catherinelam
Loves-to-Learn

I disabled selinux, fapolicyd, and firewalld, but it still happens. Although, I think we may have narrowed it down to an in-house script that runs nightly! Thanks for the help! 

0 Karma

phanTom
SplunkTrust
SplunkTrust

@catherinelam I have not seen this before but it does look Postgres-ey. 

Is this a single instance or Hot/Warm standby? If so are you sure the postgres stream is allowed (5432) between them and you have confirmed the sync is working?

The files are definitely Postgres files but I am not sure what action creates them and why they would be deleted during runtime to then "go missing". 

I hope you have also raised a support case for this too?

 

0 Karma

catherinelam
Loves-to-Learn

This is a warm standby, and the primary and warm standby show the same behaviour. 

Additionally, we have some standalone servers that also show it, so I don't think it's specific to a certain architecture. 

I tried opening a support case ticket, but whenever I submit a ticket I just get a blank page and it doesn't go through 😞 I've reached out to a company contact to see if I can escalate the issue. 

Thanks for looking!

0 Karma
Get Updates on the Splunk Community!

Database Performance Sidebar Panel Now on APM Database Query Performance & Service ...

We’ve streamlined the troubleshooting experience for database-related service issues by adding a database ...

IM Landing Page Filter - Now Available

We’ve added the capability for you to filter across the summary details on the main Infrastructure Monitoring ...

Dynamic Links from Alerts to IM Navigators - New in Observability Cloud

Splunk continues to improve the troubleshooting experience in Observability Cloud with this latest enhancement ...