Hi, we recently set up a Splunk SOAR cluster. On-poll from all app assets fail to create containers with below errors: Successfully added containers: 0, Containers failed to save: 1 Unable to create container: Container addition failed, reason from server: Authentication failed: Invalid authentication: Request Validation Error: Invalid or missing session token. Please refresh your session. nginx on the receiving node logs a steady stream of 401s on /rest/container and /rest/decided_list/..., all from python-requests, all originating from cluster nodes via the LB. I added a temporary log_format to nginx to capture auth headers (standard access log doesn't show them): nginx log_format debug_auth '$remote_addr [$time_local] "$request" $status '
'reqid="$request_id" '
'xff="$http_x_forwarded_for" '
'realip="$realip_remote_addr" '
'authz="$http_authorization" '
'phtok="$http_ph_auth_token" '
'cookie="$http_cookie"'; Result: 10.0.0.66 [23/Jul/2026:15:21:19 +0400] "POST /rest/container HTTP/1.1" 401
reqid="508d612e..." xff="10.0.0.48, 10.0.0.48" realip="10.0.0.66"
authz="-" phtok="-" cookie="sessionid=" So the requests carry no Authorization header, no ph-auth-token header, and an empty sessionid cookie. Occasionally the cookie is sessionid=None; sessionid= — i.e. a stringified null session. The 401 is therefore correct server behaviour; nothing is being presented to authenticate with. Root cause appears to be upstream of that /opt/phantom/var/log/phantom/ingestd.log shows the daemon failing to obtain a session in the first place, on a 5-minute cadence: WARNING: LIBPHIPC : ipc.cpp : 523 : IPC for REST ENDPOINT: get_session:
client connect(to port: 8443) call failed for socket 20. Error: Connection refused
WARNING: LIBPHREST : user_session.cpp : 88 : Failed to connect on port 8443
WARNING: INGESTD : ingestor_worker.cpp : 474 : Existing child_spawn_pid value: 0. Expected to be -1
WARNING: LIBPHREST : user_session.cpp : 127 : No user session token specified for closing user session get_session is refused, so the daemon has no token, and then proceeds to POST the container anyway with empty credentials — producing the 401 above. What has been ruled out nginx is up and bound on both stacks. ss -tlnp shows 0.0.0.0:8443 and [::]:8443 with live worker PIDs at the time the refusals were logged. Database is healthy. psql through pgbouncer on 6432 returns immediately; pgbouncer log shows normal traffic to the backend. (Older pgbouncer cannot connect to server entries in the logs were a month stale and unrelated.) Platform services are up. supervisorctl status (via -c /opt/phantom/etc/supervisord.conf, socket at /opt/phantom/tmp/supervisor.sock, not TCP 9001) shows rabbitmq, consul, celery workers, scheduled, add_to_es_index, broker_proxy all RUNNING with 77-day uptime. Daemons are running. phantom_ingestd, phantom_decided, phantom_workflowd, phantom_clusterd all present under phantom_watchdogd. Not stale daemon state. Killed phantom_ingestd and let watchdogd respawn it — new PID reproduces the identical Connection refused on the very next cycle. Disk is fine (23% on /, 7% on /var). Why is get_session to port 8443 refused from the local daemon while nginx is verifiably listening on 8443 on both IPv4 and IPv6? When we manually test on poll action, containers are created. We suspect automation user is used by ingestd and the failing sessions should be that user's. We gave all roles and tried regenerating its token.
... View more
