Hello All
When I open the Splunk App for Windows Infrastructure, I get a Prerequisites message: "Key value store must be enabled"
I already deleted the mongo lock file and grant 400 permissions to apps/splunk/var/lib/splunk/kvstore/mongo/splunk.key
and genenrated ssl with 1024
Any ideas what to check next?
Tnx in advance!
Sounds like your firewall ports are not open. Run netstat -an
and look for something like "LISTENING" on 127.0.0.1:(YOUR_KVSTORE_PORT).
If you're "listening" on that port, great, kvstore is enabled and splunk bound to the port successfully upon startup. Now you must test connectivity to that server on that port. An easy way to do this is with telnet.
Another method is using the nc
command in linux (netcat).
With splunk shut down, on the server with issues do the following:
1. stop splunk
2. Tell netcat to listen on your kvstore port (binds to the port) nc -l {YOUR_KVSTORE_PORT}
3. From another server with network connectivity to the server in question: nc splunkserver {YOUR_KVSTORE_PORT}
example: `nc splunksearchheadip 8191'
4. Both servers will have a blank screen with blinking cursor, now if you type on the terminal on the remote server, you should see words printed on the "problematic" server. This will prove that network connectivity on your kvstore port exists. If this test fails, you will not see anything printed on the screen of the problematic server. That would meant the port isnt open.
Hi
I see this error in mongo.log
Did not find local replica set configuration document at startup; NoMatchingDocument Did not find replica set configuration document in local.system.replset
Hi,
Tnx for the reply
with splunk running service i see that port 8191 is listening and telnet works
right! i tested it from own server
Do you have a search head cluster or search head pool? If so, can they telnet to the others just fine on the same kv store port? If it's more than one search head, you shouldnt be testing telnet from the local machine to the local machine, but making it traverse the network path between two machines instead.
When you say telnet works, you mean you get a blank screen with a blinking cursor?
What error messages are logged in splunkd.log when you open the app? Did you verify that server.conf has the kvstore enabled, i.e. you see disabled=false under the [kvstore] stanza?
Hi on other lab server it is enabled , i changed it in
/opt/splunk/etc/system/default/server.conf , kvstore was disabled in one section to true
[introspection:generator:kvstore]
disabled = false (was true)
but on other with same settiings i am get error that is not enabled.
Maybe mongo db is corrupted and needs to be repaired?
Tnx
Hi,
I see the kvstore is enabled (disabled=false)
Hello
tnx for reply
I already configured 700 permissions then reverted back to 400 with no luck.
Take a look at RichING's comments.
https://answers.splunk.com/answers/268584/splunk-app-for-windows-infrastructure-it-has-error.html
And 2nd EDIT from juriggs
https://answers.splunk.com/answers/206030/splunk-app-for-windows-infrastructure-why-am-i-get-1.html
its not permissions issue i think , maybe mongo db corrupted?
I can't tell mate, you may need help from support team.
Hi there mate,
Try this from a shell and then restart Splunk.
chmod 700 /opt/splunk/var/lib/splunk/kvstore/mongo/splunk.key