splunkd crashes on startup with "ERROR SQLitePersistentStorageImpl - Error executing: create table keyvaluepairs_t(primarykey text constraint primary_c unique,secondary1 text, secondary2 text, secondary3 text, value text); Msg=database is locked"

[rphillips_splk]

Splunkd will not start and splunkd.log shows this error message:

"ERROR SQLitePersistentStorageImpl - Error executing: create table keyvaluepairs_t(primarykey text constraint primary_c unique,secondary1 text, secondary2 text, secondary3 text, value text); Msg=database is locked"

environment:
Splunk 6.4.1
Linux
Splunk install sits on an NFS
Splunk indexed data sits on iscsi

[rphillips_splk · Accepted solution]

In this case the nfslock daemon was not running on the nfs server (10.10.10.25).

on the indexer the /var/log/messages file shows errors like:
kernel: lockd: cannot monitor 10.10.10.25

• not all indexers exhibited this problem because they were using different nfs servers for the splunk install
• to show what the nfs mount config looks like run 'cat /etc/fstab'

To resolve this issue:
start the nfslock service on the NFS server.
Delete $SPLUNK_HOME/var/lib/splunk/persistentstorage/seqno_db on the Splunk instance.
Start Splunk