After upgrading from Splunk Enterprise 9.2.2 to 9.2.4, the following error is displayed in the Splunk Web message:

[shiba]

After upgrading from Splunk Enterprise 9.2.2 to 9.2.4, the following error is displayed in the Splunk Web message:

After upgrading Splunk Enterprise from 9.2.2 to 9.2.4, the following error message started appearing on Splunk Web.
Log collection and searching is possible.

A-Server acts as an indexer, and one search and indexer are used.

Search peer A-Server has the following message: Failed to start KV Store process. See mongod.log and splunkd.log for details.
2024/12/25 11:34:12
Search peer A-Server has the following message: KV Store changed status to failed. KVStore process terminated..
2024/12/25 11:34:11
Search peer A-Server has the following message: KV Store process terminated abnormally (exit code 14, status PID 29873 exited with code 14). See mongod.log and splunkd.log for details.
2024/12/25 11:34:11
Search peer A-Server has the following message: Security risk warning: Found an empty value for 'allowedDomainList' in the alert_actions.conf configuration file. If you do not configure this setting, then users can send email alerts with search results to any domain. You can add values for 'allowedDomainList' either in the alert_actions.conf file or in Server Settings > Email Settings > Email Domains in Splunk Web.
2024/12/25 11:34:11
Failed to start KV Store process. See mongod.log and splunkd.log for details.
2024/12/25 11:26:57
Security risk warning: Found an empty value for 'allowedDomainList' in the alert_actions.conf configuration file. If you do not configure this setting, then users can send email alerts with search results to any domain. You can add values for 'allowedDomainList' either in the alert_actions.conf file or in Server Settings > Email Settings > Email Domains in Splunk Web.
2024/12/25 11:26:57
KV Store changed status to failed. KVStore process terminated..
2024/12/25 11:26:56
KV Store process terminated abnormally (exit code 14, status PID 2757 exited with code 14). See mongod.log and splunkd.log for details.
2024/12/25 11:26:56

[kiran_panchavat]

@shiba KV Store issues usually occur when Splunk's Key-Value Store is not functioning properly, which can impact searches that depend on KV Store collections. But if you are getting this ERROR on indexers. Ignore it and "Security Risk Warning: Found an Empty Value for 'allowedDomainList':-The allowedDomainList parameter in alert_actions.conf is not configured properly, leaving it empty. This parameter specifies the domains allowed for sending alerts (e.g., via email). 

If this server is being used as both an indexer and a search head, please confirm.

[richgalloway]

All of those messages can be ignored. 

Server-A is an indexer and indexers do not use the KVStore.  In fact, KVStore can be disabled on indexers.

The "Found an empty value for 'allowedDomainList' " message can be ignored if you choose to, especially on an indexer.  If you're concerned about security and the sending of emails outside certain domains then follow the instructions in the message.

[shiba]

Hello, thanks for your answer.
I understand about the indexer.
Is there any problem with the following message?

Failed to start KV Store process. See mongod.log and splunkd.log for details.
2024/12/25 11:26:57
Security risk warning: Found an empty value for 'allowedDomainList' in the alert_actions.conf configuration file. If you do not configure this setting, then users can send email alerts with search results to any domain. You can add values for 'allowedDomainList' either in the alert_actions.conf file or in Server Settings > Email Settings > Email Domains in Splunk Web.
2024/12/25 11:26:57
KV Store changed status to failed. KVStore process terminated..
2024/12/25 11:26:56
KV Store process terminated abnormally (exit code 14, status PID 2757 exited with code 14). See mongod.log and splunkd.log for details.
2024/12/25 11:26:56

[richgalloway · Accepted solution]

---

@shiba wrote:

Security risk warning: Found an empty value for 'allowedDomainList' in the alert_actions.conf configuration file. If you do not configure this setting, then users can send email alerts with search results to any domain. You can add values for 'allowedDomainList' either in the alert_actions.conf file or in Server Settings > Email Settings > Email Domains in Splunk Web.

As already explained, this warning matters only if you care about where alert emails can be sent.

---

Failed to start KV Store process. See mongod.log and splunkd.log for details.
2024/12/25 11:26:57
KV Store changed status to failed. KVStore process terminated..
2024/12/25 11:26:56
KV Store process terminated abnormally (exit code 14, status PID 2757 exited with code 14). See mongod.log and splunkd.log for details.
2024/12/25 11:26:56

---

These messages definitely are a problem on a search head, but not on an indexer.  Consult mongod.log for details about the problem and fix what is reported.  For indexers, turn off KVStore by adding the following to server.conf

[kvstore]
disabled=true