Monitoring Splunk

splunkd crashes on startup with "ERROR SQLitePersistentStorageImpl - Error executing: create table keyvaluepairs_t(primarykey text constraint primary_c unique,secondary1 text, secondary2 text, secondary3 text, value text); Msg=database is locked"

rphillips_splk
Splunk Employee
Splunk Employee

Splunkd will not start and splunkd.log shows this error message:

"ERROR SQLitePersistentStorageImpl - Error executing: create table keyvaluepairs_t(primarykey text constraint primary_c unique,secondary1 text, secondary2 text, secondary3 text, value text); Msg=database is locked"

environment:
Splunk 6.4.1
Linux
Splunk install sits on an NFS
Splunk indexed data sits on iscsi

0 Karma
1 Solution

rphillips_splk
Splunk Employee
Splunk Employee

In this case the nfslock daemon was not running on the nfs server (10.10.10.25).

on the indexer the /var/log/messages file shows errors like:
kernel: lockd: cannot monitor 10.10.10.25

  • not all indexers exhibited this problem because they were using different nfs servers for the splunk install
  • to show what the nfs mount config looks like run 'cat /etc/fstab'

To resolve this issue:
start the nfslock service on the NFS server.
Delete $SPLUNK_HOME/var/lib/splunk/persistentstorage/seqno_db on the Splunk instance.
Start Splunk

View solution in original post

rphillips_splk
Splunk Employee
Splunk Employee

In this case the nfslock daemon was not running on the nfs server (10.10.10.25).

on the indexer the /var/log/messages file shows errors like:
kernel: lockd: cannot monitor 10.10.10.25

  • not all indexers exhibited this problem because they were using different nfs servers for the splunk install
  • to show what the nfs mount config looks like run 'cat /etc/fstab'

To resolve this issue:
start the nfslock service on the NFS server.
Delete $SPLUNK_HOME/var/lib/splunk/persistentstorage/seqno_db on the Splunk instance.
Start Splunk

Get Updates on the Splunk Community!

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...

March Community Office Hours Security Series Uncovered!

Hello Splunk Community! In March, Splunk Community Office Hours spotlighted our fabulous Splunk Threat ...

Stay Connected: Your Guide to April Tech Talks, Office Hours, and Webinars!

Take a look below to explore our upcoming Community Office Hours, Tech Talks, and Webinars in April. This post ...