Why are our notables are not visible in the Splunk Enterprise Security Incident review dashboard?

[srisahitya_v]

Dear Team,

Splunk version - 7.0.1
ES version - 4.7.4

1. After patching activity, i have rebooted the Splunk cluster servers, ES search heads, Indexers and clustermaster, deployment server.

2. After rebooting, ES is not showcasing Notables. and it is not even showcasing results for notables macro.

3. Error coming as "Error in 'lookup' command: Lookups: The lookup table 'notable_xref_lookup' does not exist or is not available."
   query: $SPLUNK_HOME/bin/splunk search "| rest /services/server/info splunk_server=* | fields splunk_server, kvStoreStatus"

I have checked the clustermaster for kvstore results:
clustermaster failed
deployment server failed
deployeres1 failed
non-es failed
EnterPrise security Search Head Instances are-------- starting
idx 1,2,3,4,5,6,7 ready

Can any one help me here.

[shirishkamat84 · Accepted solution]

This was previously mentioned in a different post. Here is what you need to know:

check the expiry of the existing server.pem file, run the below command
$SPLUNK_HOME/bin/splunk cmd openssl x509 -enddate -noout -in /opt/splunk/etc/auth/server.pem

if the certificate is expired, then generate a new one which fixed the issue for us.

Generate the new cert:
mv $SPLUNK_HOME/etc/auth/server.pem $SPLUNK_HOME/etc/auth/server.pem.old

restart Splunk and this should fix the issue. Run this on the instance where KV store is being used.

[harsmarvania57]

Hi,

notable macro is depend on notable_xref_lookup and notable_xref_lookup is KVStore lookup so you need to wait till your KVStore will be UP on ES Search Head Instances. If KVStore took too much time start then you need to check $SPLUNK_HOME/var/log/splunk/splunkd.log , $SPLUNK_HOME/var/log/splunk/mongod.log and $SPLUNK_HOME/var/log/introspection/kvstore.log & check why KVStore is taking more time to start.