Conflict with those parameters causes enviroment with Splunk Enterprise Security to overflow kvstore with too many connections and thus preventing getting results from KV store. ### Environment - Product / Version: Splunk Enterprise 10.6 (RHEL) - Apps: Splunk Enterprise Security (ES) - KV Store Engine: defaultKVStoreType = cohosted (PDL / splunk-postgres sidecar) - Hardware: 46 GB RAM (27 GB available), low disk I/O wait (vmstat wa = 0) - KV Store DB size: ~68 GB ### Problem Summary After migrating to Splunk 10.6 with the cohosted KV Store engine, `kvstore-pdl` enters a recurring ~6-minute crash/restart loop triggered by `splunk-supervisor` health check failures (HTTP 500). During these loops, scheduled and ad-hoc lookups fail with `External lookup table ... returned error code 0`. The issue persists across full Splunk and OS reboots. ### Root Cause Identified There is a default configuration mismatch between PgBouncer and PostgreSQL in the `splunk-postgres` package: - `pgbouncer.ini` sets `default_pool_size = 10` (`pool_mode = transaction`, `max_client_conn = 256`) - PostgreSQL sets `datconnlimit = 5` on the `kvstore` database (`max_connections = 100`, `shared_buffers = 128MB`) We verified this exact default configuration on a second, freshly migrated Splunk 10.6 instance (without ES, `kvstore` DB size 113 MB), which also ships with `kvstore` `datconnlimit = 5` and PgBouncer `default_pool_size = 10`: datname | datconnlimit | size ---------------------+--------------+-------- kvstore | 5 | 113 MB enterprise_security | 10 | 7494 kB ### Key Log Excerpts 1. postgres-*.log: 2026-11-19 08:09:04.549 CET [26916] FATAL: too many connections for database "kvstore" 2026-11-19 08:09:19.694 CET [27069] FATAL: too many connections for database "kvstore" 2. pgbouncer.log: 2026-11-19 08:09:04.549 CET [7834] WARNING server login failed: FATAL too many connections for database "kvstore" 2026-11-19 08:09:04.549 CET [7834] LOG S-0x555d01271ad8: kvstore/
[email protected]:5432 closing because: too many connections for database "kvstore" (age=0s) 3. sup-pkg-kvstore-pdl-stdout.log: {"level":"INFO","operation":"checkTenantDBProvision","status":"failure","error":"driver: bad connection"} {"level":"ERROR","location":"kvstore-pdl/health_check.go:163","message":"Ping failed","component":"PdlHealthCheck","status":"unhealthy","error":"internal server error"} 4. supervisor.log: {"level":"ERROR","location":"health/monitor.go:82","message":"Health Check has failed","sidecarID":"kvstore-pdl-...","error":"response had 500 status, response body: Health check failure threshold reached, failed count= 7, allowed count=3\n"} {"level":"INFO","message":"Process returned with value","process":"kvstore-pdl (v0.0.1)","error":"signal: killed"} ### Temporary Workaround Applied Without modifying any Splunk or Postgres config files on disk, we ran: - `ALTER DATABASE kvstore CONNECTION LIMIT 50;` - `ANALYZE;` on the `kvstore` database This immediately eliminated the `FATAL: too many connections` errors, stopped the `kvstore-pdl` restart loop, and restored lookup functionality. ### Questions 1. Is the mismatch between `datconnlimit = 5` on the `kvstore` database and `default_pool_size = 10` in PgBouncer a known defect in Splunk 10.6 cohosted KV Store? If so, could you share the bug ID and target fix? Thanks
... View more