Monitoring Splunk

splunkd crashes on startup with "ERROR SQLitePersistentStorageImpl - Error executing: create table keyvaluepairs_t(primarykey text constraint primary_c unique,secondary1 text, secondary2 text, secondary3 text, value text); Msg=database is locked"

rphillips_splk
Splunk Employee
Splunk Employee

Splunkd will not start and splunkd.log shows this error message:

"ERROR SQLitePersistentStorageImpl - Error executing: create table keyvaluepairs_t(primarykey text constraint primary_c unique,secondary1 text, secondary2 text, secondary3 text, value text); Msg=database is locked"

environment:
Splunk 6.4.1
Linux
Splunk install sits on an NFS
Splunk indexed data sits on iscsi

0 Karma
1 Solution

rphillips_splk
Splunk Employee
Splunk Employee

In this case the nfslock daemon was not running on the nfs server (10.10.10.25).

on the indexer the /var/log/messages file shows errors like:
kernel: lockd: cannot monitor 10.10.10.25

  • not all indexers exhibited this problem because they were using different nfs servers for the splunk install
  • to show what the nfs mount config looks like run 'cat /etc/fstab'

To resolve this issue:
start the nfslock service on the NFS server.
Delete $SPLUNK_HOME/var/lib/splunk/persistentstorage/seqno_db on the Splunk instance.
Start Splunk

View solution in original post

rphillips_splk
Splunk Employee
Splunk Employee

In this case the nfslock daemon was not running on the nfs server (10.10.10.25).

on the indexer the /var/log/messages file shows errors like:
kernel: lockd: cannot monitor 10.10.10.25

  • not all indexers exhibited this problem because they were using different nfs servers for the splunk install
  • to show what the nfs mount config looks like run 'cat /etc/fstab'

To resolve this issue:
start the nfslock service on the NFS server.
Delete $SPLUNK_HOME/var/lib/splunk/persistentstorage/seqno_db on the Splunk instance.
Start Splunk

Get Updates on the Splunk Community!

Synthetic Monitoring: Not your Grandma’s Polyester! Tech Talk: DevOps Edition

Register today and join TekStream on Tuesday, February 28 at 11am PT/2pm ET for a demonstration of Splunk ...

Instrumenting Java Websocket Messaging

Instrumenting Java Websocket MessagingThis article is a code-based discussion of passing OpenTelemetry trace ...

Announcing General Availability of Splunk Incident Intelligence!

Digital transformation is real! Across industries, companies big and small are going through rapid digital ...