How to fix postgres sidecar collation errors

[las]

Hi.

I recently upgraded my Deployment Server and thought I would look into the Edge Processor.

Unfortunately it is not very stable in my environment, and I see a lot of errors:

2026-06-10 16:45:57.523 CEST [53166] WARNING:  database "acies_config_service" has a collation version mismatch
2026-06-10 16:45:57.523 CEST [53166] DETAIL:  The database was created using collation version 2.28, but the operating system provides version 2.34.
2026-06-10 16:45:57.523 CEST [53166] HINT:  Rebuild all objects in this database that use the default collation and run ALTER DATABASE acies_config_service REFRESH COLLATION VERSION, or build PostgreSQL with the right library version.

 error: error creating database -- Schema file for setting up database for sidecar service
--
-- Version: 1.3
-- Date: 02/09/2026
--
-- Change Log:
-- Version 1.3 (02/09/2026):
--   - Added database for ep_spl2_preview service.
-- Version 1.2 (02/26/2026):
--   - Added database for agent_manager service.
-- Version 1.1 (09/02/2025):
--   - Updated postgres server side per db connection limit for all databases.
-- Version 1.0 (05/17/2024):
--   - Initial schema creation for search_metadata, kvstore, acies_config_service, and opamp_service.

CREATE DATABASE search_metadata WITH OWNER = "do.root.splunk" CONNECTION LIMIT 10;
CREATE DATABASE kvstore WITH OWNER = "kvstore.pdl.splunk" TEMPLATE = template0 LC_COLLATE = 'C' LC_CTYPE = 'C' CONNECTION LIMIT 20;
CREATE DATABASE acies_config_service WITH OWNER = "acc.dbuser.splunk" CONNECTION LIMIT 10;
CREATE DATABASE opamp_service WITH OWNER = "oas.root.splunk" CONNECTION LIMIT 10;
CREATE DATABASE agent_manager WITH OWNER = "am.root.splunk" CONNECTION LIMIT 10;
CREATE DATABASE ep_spl2_preview WITH OWNER = "ep_preview.root.splunk" CONNECTION LIMIT 10;
: pq: template database "template1" has a collation version mismatch
	Failed to create database
	failed to create database
	Failed to bootstrap client artifact
   log_level: ERROR
   msg: Failed to bootstrap databases in standalone 

I have found a lot of suggestions that the databases shoul be rebuild - from splunk.my.site.com/customer/s/article/database-has-a-collation-version-mismatch for instance, where they give these instructions:

psql -h /tmp -p 6432 -d postgres -c "ALTER DATABASE postgres REFRESH COLLATION VERSION;"
psql -h /tmp -p 6432 -d postgres -c "REINDEX DATABASE postgres;"
psql -h /tmp -p 6432 -d phantom -c "ALTER DATABASE phantom REFRESH COLLATION VERSION;"
psql -h /tmp -p 6432 -d phantom -c "REINDEX DATABASE phantom;"

Now I'm unsure how to run these commands (I do not have root nor sudo for anythin outside /opt/splunk and even there limited) and exactly for which databases should these commands be run.

Hopefully someone out here has an answer, otherwise I'll create a case and post their answer.

Kind regards

las

[las · Accepted solution]

I got a response from Support, that solved my problem.

A: Recreating the exisiting DB:

1. Navigate to /opt/splunk/var/packages/data/postgres/
2. Rename db folder as db.backup
3. Restart Splunk.

B: If above does not help:

1. Connect to template1
     \c template1
     ALTER DATABASE template1 REFRESH COLLATION VERSION;

2. Connect to postgres
     \c postgres
    ALTER DATABASE postgres REFRESH COLLATION VERSION;

I only needed to perform step A

[tscroggins]

Hi @las,

The article you referenced is for Splunk SOAR, but if you want to explore the PostgreSQL environment shipped with Splunk Enterprise, you can install PostgreSQL 17 tools on your Linux server.

Make a backup of your Splunk Enterprise installation before proceeding. I recommend using a restored backup running in a sandbox. I wouldn't do any of this on a production deployment server.

This process is for Red Hat Enterprise Linux 10. It's similar for other distributions. PostgreSQL publishes a handy guide at https://www.postgresql.org/download/linux/.

1. Disable the RHEL postgresql module:

sudo dnf module disable postgresql

2. Install the PostgreSQL repository:

sudo dnf install https://download.postgresql.org/pub/repos/yum/reporpms/EL-10-x86_64/pgdg-redhat-repo-latest.noarch.rpm

3. Install PostgreSQL 17 tools:

sudo dnf install postgresql17

4. Connect to PostgreSQL:

psql "host=localhost port=5432 dbname=postgres user=postgres_admin"

Your local postgres_admin password is encrypted in $SPLUNK_HOME/etc/system/local/passwords.conf.

You can decrypt the value with:

$SPLUNK_HOME/bin/splunk show-decrypted --value '<password>'

where <password> is the encrypted value in passwords.conf.

Or all together:

psql "host=localhost port=5432 dbname=postgres user=postgres_admin password=$($SPLUNK_HOME/bin/splunk show-decrypted --value "$($SPLUNK_HOME/bin/splunk btool passwords list credential:postgres:postgres_admin: | grep '^password\s*=' | sed 's/^password[[:space:]]*=[[:space:]]*//')")"

From here, you can list databases with \l, list users with \du, or run other commands as needed.

Can you damage the installation beyond repair? YES. Verify your backup before making changes.

It would be best to proceed with the help of Splunk support. If the issue you have is reproducible, they can fix it for other customers, too.

[las]

Thanks tscroggins, I better activate Support.