Monitoring Splunk

splunkd crashes on startup with "ERROR SQLitePersistentStorageImpl - Error executing: create table keyvaluepairs_t(primarykey text constraint primary_c unique,secondary1 text, secondary2 text, secondary3 text, value text); Msg=database is locked"

rphillips_splk
Splunk Employee
Splunk Employee

Splunkd will not start and splunkd.log shows this error message:

"ERROR SQLitePersistentStorageImpl - Error executing: create table keyvaluepairs_t(primarykey text constraint primary_c unique,secondary1 text, secondary2 text, secondary3 text, value text); Msg=database is locked"

environment:
Splunk 6.4.1
Linux
Splunk install sits on an NFS
Splunk indexed data sits on iscsi

0 Karma
1 Solution

rphillips_splk
Splunk Employee
Splunk Employee

In this case the nfslock daemon was not running on the nfs server (10.10.10.25).

on the indexer the /var/log/messages file shows errors like:
kernel: lockd: cannot monitor 10.10.10.25

  • not all indexers exhibited this problem because they were using different nfs servers for the splunk install
  • to show what the nfs mount config looks like run 'cat /etc/fstab'

To resolve this issue:
start the nfslock service on the NFS server.
Delete $SPLUNK_HOME/var/lib/splunk/persistentstorage/seqno_db on the Splunk instance.
Start Splunk

View solution in original post

rphillips_splk
Splunk Employee
Splunk Employee

In this case the nfslock daemon was not running on the nfs server (10.10.10.25).

on the indexer the /var/log/messages file shows errors like:
kernel: lockd: cannot monitor 10.10.10.25

  • not all indexers exhibited this problem because they were using different nfs servers for the splunk install
  • to show what the nfs mount config looks like run 'cat /etc/fstab'

To resolve this issue:
start the nfslock service on the NFS server.
Delete $SPLUNK_HOME/var/lib/splunk/persistentstorage/seqno_db on the Splunk instance.
Start Splunk

Get Updates on the Splunk Community!

The Splunk Success Framework: Your Guide to Successful Splunk Implementations

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...

Splunk Training for All: Meet Aspiring Cybersecurity Analyst, Marc Alicea

Splunk Education believes in the value of training and certification in today’s rapidly-changing data-driven ...

Investigate Security and Threat Detection with VirusTotal and Splunk Integration

As security threats and their complexities surge, security analysts deal with increased challenges and ...