Hi, my 10.0.3 systems (SH, DS and indexers) send me several hundred postgres errors each day that seem to indicate that there is an initialization issue:
IDX & SH:
{"time":"2026-03-04T09:23:53.710005158+01:00","level":"ERROR","source":{"function":"postgres-service/package/teleportplugin.(*PostgresPlugin).BootstrapPostgres","file":"postgres-service/package/teleportplugin/postgres.go","line":529},"msg":"Failed to setup databases and users:","Err":"error creating database -- Schema file for setting up database for sidecar service\n-- Version: 1.0\n-- Date: 05/17/2024\n\nCREATE DATABASE search_metadata WITH OWNER = \"do.root.splunk\" CONNECTION LIMIT 5;\nCREATE DATABASE kvstore WITH OWNER = \"kvstore.pdl.splunk\" TEMPLATE = template0 LC_COLLATE = 'C' LC_CTYPE = 'C' CONNECTION LIMIT 5;\nCREATE DATABASE acies_config_service WITH OWNER = \"acc.dbuser.splunk\" CONNECTION LIMIT 5;\nCREATE DATABASE opamp_service WITH OWNER = \"oas.root.splunk\" CONNECTION LIMIT 5;\n: pq: template database \"template1\" has a collation version mismatch\n\tFailed to create database\n\tfailed to create database"}DS:
{"time":"2026-03-04T09:26:19.466646819+01:00","level":"ERROR","source":{"function":"postgres-service/package/teleportplugin.(*PostgresPlugin).BootstrapPostgres","file":"postgres-service/package/teleportplugin/postgres.go","line":521},"msg":"Failed to initialize DAL:","Error":"failed to ping PostgreSQL: pq: password authentication failed for user \"postgres_admin\"\n\tFailed to connect to Postgres\n\tFailed to connect to Postgres"}
{"time":"2026-03-04T09:26:19.466614486+01:00","level":"ERROR","source":{"function":"postgres-service/internal/db.NewDAL","file":"postgres-service/internal/db/dal.go","line":64},"msg":"Failed to initialize database connection","error":"failed to ping PostgreSQL: pq: password authentication failed for user \"postgres_admin\"\n\tFailed to connect to Postgres"}I am a bit lost on what I need to do to set this up.
thx
afx
You seem to have two separate issues. One is with collation versions. You might try to follow the advice from https://www.cisco.com/c/en/us/support/docs/observability/splunk-platform/splunk-enterprise/7/SPLK055...
Another seems to be similar to https://www.cisco.com/c/en/us/support/docs/observability/splunk-platform/splunk-data-stream-processo...
PS, also theck the
RemoveIPC=no
i.e. do not automatically remove a user's System V IPC objects and POSIX message queues when that user fully logs out
Two cents:
1. ensure chmod 600 /opt/splunk/etc/auth/server.pem
2. check if 127.0.0.1 is resolvable on the machine (in etc/hosts it can be commented in favor of ::1) as the internal postgres traefik proxy has a hardcoded 127.0.0.1 dependency