Monitoring Splunk

splunkd crashes on startup with "ERROR SQLitePersistentStorageImpl - Error executing: create table keyvaluepairs_t(primarykey text constraint primary_c unique,secondary1 text, secondary2 text, secondary3 text, value text); Msg=database is locked"

rphillips_splk
Splunk Employee
Splunk Employee

Splunkd will not start and splunkd.log shows this error message:

"ERROR SQLitePersistentStorageImpl - Error executing: create table keyvaluepairs_t(primarykey text constraint primary_c unique,secondary1 text, secondary2 text, secondary3 text, value text); Msg=database is locked"

environment:
Splunk 6.4.1
Linux
Splunk install sits on an NFS
Splunk indexed data sits on iscsi

0 Karma
1 Solution

rphillips_splk
Splunk Employee
Splunk Employee

In this case the nfslock daemon was not running on the nfs server (10.10.10.25).

on the indexer the /var/log/messages file shows errors like:
kernel: lockd: cannot monitor 10.10.10.25

  • not all indexers exhibited this problem because they were using different nfs servers for the splunk install
  • to show what the nfs mount config looks like run 'cat /etc/fstab'

To resolve this issue:
start the nfslock service on the NFS server.
Delete $SPLUNK_HOME/var/lib/splunk/persistentstorage/seqno_db on the Splunk instance.
Start Splunk

View solution in original post

rphillips_splk
Splunk Employee
Splunk Employee

In this case the nfslock daemon was not running on the nfs server (10.10.10.25).

on the indexer the /var/log/messages file shows errors like:
kernel: lockd: cannot monitor 10.10.10.25

  • not all indexers exhibited this problem because they were using different nfs servers for the splunk install
  • to show what the nfs mount config looks like run 'cat /etc/fstab'

To resolve this issue:
start the nfslock service on the NFS server.
Delete $SPLUNK_HOME/var/lib/splunk/persistentstorage/seqno_db on the Splunk instance.
Start Splunk

Get Updates on the Splunk Community!

Security Highlights | November 2022 Newsletter

 November 2022 2022 Gartner Magic Quadrant for SIEM: Splunk Named a Leader for the 9th Year in a RowSplunk is ...

Platform Highlights | November 2022 Newsletter

 November 2022 Skill Up on Splunk with our New Builder Tech Talk SeriesCan you build it? Yes you can! *play ...

Splunk Education - Fast Start Program!

Welcome to Splunk Education! Splunk training programs are designed to enable you to get started quickly and ...