kv store problem

[sarit_s]

hello
i'm running splunk with Kubernetese and Ansible
from time to time im getting this error :

[SPLUNKD] Error in 'inputlookup' command: External command based lookup
'kv_alerts_prod' is not available
because KV Store initialization has
failed. Contact your system
administrator.
{"message":"{\"response\":{\"headers\":{\"date\":\"Mon,
09 Mar 2020 07:44:04
GMT\",\"expires\":\"Thu, 26 Oct 1978
00:00:00
GMT\",\"cache-control\":\"no-store,
no-cache, must-revalidate,
max-age=0\",\"content-type\":\"application/json;
charset=UTF-8\",\"x-content-type-options\":\"nosniff\",\"content-length\":\"215\",\"vary\":\"Cookie,
Authorization\",\"connection\":\"Close\",\"set-cookie\":[\"splunkd_8089=S8owSsAcljUIFXeya8Nhkk9y^cqA^qGsZi2mnFodHbZzb51KqkZIsqrtkEp1RVvwejUi1ADnoVtJaqV859dCuoZX^WkIKg6ZWDWM_h0Ks1lhSMRXKgpZ323DKC;
Path=/; Secure; HttpOnly;
Max-Age=3600; Expires=Mon, 09 Mar 2020
08:44:04
GMT\"],\"x-frame-options\":\"SAMEORIGIN\",\"server\":\"Splunkd\"},\"statusCode\":400},\"status\":400,\"data\":{\"messages\":[{\"type\":\"FATAL\",\"text\":\"Error
in 'inputlookup' command: External
command based lookup 'kv_alerts_prod'
is not available because KV Store
initialization has failed. Contact
your system
administrator.\"}]},\"error\":null}","level":"ERROR","logger":"argus:aviation-splunk-rest-apis:services:splunkService","timestamp":"2020-03-09T07:44:04.451Z"}
{"message":"{\"response\":{\"headers\":{\"date\":\"Mon,
09 Mar 2020 07:44:04
GMT\",\"expires\":\"Thu, 26 Oct 1978
00:00:00
GMT\",\"cache-control\":\"no-store,
no-cache, must-revalidate,
max-age=0\",\"content-type\":\"application/json;
charset=UTF-8\",\"x-content-type-options\":\"nosniff\",\"content-length\":\"215\",\"vary\":\"Cookie,
Authorization\",\"connection\":\"Close\",\"set-cookie\":[\"splunkd_8089=S8owSsAcljUIFXeya8Nhkk9y^cqA^qGsZi2mnFodHbZzb51KqkZIsqrtkEp1RVvwejUi1ADnoVtJaqV859dCuoZX^WkIKg6ZWDWM_h0Ks1lhSMRXKgpZ323DKC;
Path=/; Secure; HttpOnly;
Max-Age=3600; Expires=Mon, 09 Mar 2020
08:44:04
GMT\"],\"x-frame-options\":\"SAMEORIGIN\",\"server\":\"Splunkd\"},\"statusCode\":400},\"status\":400,\"data\":{\"messages\":[{\"type\":\"FATAL\",\"text\":\"Error
in 'inputlookup' command: External
command based lookup 'kv_alerts_prod'
is not available because KV Store
initialization has failed. Contact
your system
administrator.\"}]},\"error\":null}","level":"ERROR","logger":"argus:aviation-splunk-rest-apis","timestamp":"2020-03-09T07:44:04.451Z"}

KV Store process terminated abnormally
(exit code 100, status exited with
code 100). See mongod.log and
splunkd.log for details.

removing mongod.lock fix the problem but it is happening again . im wondering if there is another way to solve it

thanks !

[manjunathmeti]

hi @sarit_s,

Check the permission of mongo directory: $SPLUNK_HOME/var/lib/splunk/kvstore/mongo/. You can change user and group to splunk.

chown -R splunk:splunk $SPLUNK_HOME

Change file permissions for splunk.key to 600.

chmod 600 $SPLUNK_HOME/var/lib/splunk/kvstore/mongo/splunk.key

[sarit_s]

Hey
thanks for your answer
it is already set up like this