All Apps and Add-ons

Splunk App for Windows Infrastructure: Why a I getting prerequisites message "Key value store must be enabled"?

vad34
Path Finder

Hello All

When I open the Splunk App for Windows Infrastructure, I get a Prerequisites message: "Key value store must be enabled"

I already deleted the mongo lock file and grant 400 permissions to apps/splunk/var/lib/splunk/kvstore/mongo/splunk.key
and genenrated ssl with 1024

Any ideas what to check next?
Tnx in advance!

0 Karma

jkat54
SplunkTrust
SplunkTrust

Sounds like your firewall ports are not open. Run netstat -an and look for something like "LISTENING" on 127.0.0.1:(YOUR_KVSTORE_PORT).

If you're "listening" on that port, great, kvstore is enabled and splunk bound to the port successfully upon startup. Now you must test connectivity to that server on that port. An easy way to do this is with telnet.

Another method is using the nc command in linux (netcat).

With splunk shut down, on the server with issues do the following:
1. stop splunk
2. Tell netcat to listen on your kvstore port (binds to the port) nc -l {YOUR_KVSTORE_PORT}
3. From another server with network connectivity to the server in question: nc splunkserver {YOUR_KVSTORE_PORT} example: `nc splunksearchheadip 8191'
4. Both servers will have a blank screen with blinking cursor, now if you type on the terminal on the remote server, you should see words printed on the "problematic" server. This will prove that network connectivity on your kvstore port exists. If this test fails, you will not see anything printed on the screen of the problematic server. That would meant the port isnt open.

0 Karma

vad34
Path Finder

Hi
I see this error in mongo.log

Did not find local replica set configuration document at startup; NoMatchingDocument Did not find replica set configuration document in local.system.replset

0 Karma

vad34
Path Finder

Hi,
Tnx for the reply
with splunk running service i see that port 8191 is listening and telnet works

0 Karma

vad34
Path Finder

right! i tested it from own server

0 Karma

jkat54
SplunkTrust
SplunkTrust

Do you have a search head cluster or search head pool? If so, can they telnet to the others just fine on the same kv store port? If it's more than one search head, you shouldnt be testing telnet from the local machine to the local machine, but making it traverse the network path between two machines instead.

0 Karma

jkat54
SplunkTrust
SplunkTrust

When you say telnet works, you mean you get a blank screen with a blinking cursor?

0 Karma

s2_splunk
Splunk Employee
Splunk Employee

What error messages are logged in splunkd.log when you open the app? Did you verify that server.conf has the kvstore enabled, i.e. you see disabled=false under the [kvstore] stanza?

0 Karma

vad34
Path Finder

Hi on other lab server it is enabled , i changed it in
/opt/splunk/etc/system/default/server.conf , kvstore was disabled in one section to true
[introspection:generator:kvstore]
disabled = false (was true)
but on other with same settiings i am get error that is not enabled.
Maybe mongo db is corrupted and needs to be repaired?
Tnx

0 Karma

vad34
Path Finder

Hi,
I see the kvstore is enabled (disabled=false)

0 Karma

vad34
Path Finder

Hello
tnx for reply
I already configured 700 permissions then reverted back to 400 with no luck.

0 Karma

alemarzu
Motivator
0 Karma

vad34
Path Finder

its not permissions issue i think , maybe mongo db corrupted?

0 Karma

alemarzu
Motivator

I can't tell mate, you may need help from support team.

0 Karma

alemarzu
Motivator

Hi there mate,

Try this from a shell and then restart Splunk.
chmod 700 /opt/splunk/var/lib/splunk/kvstore/mongo/splunk.key

0 Karma
Take the 2021 Splunk Career Survey

Help us learn about how Splunk has
impacted your career by taking the 2021 Splunk Career Survey.

Earn $50 in Amazon cash!