Splunk Enterprise Security

Why are our notables are not visible in the Splunk Enterprise Security Incident review dashboard?

srisahitya_v
Communicator

Dear Team,

Splunk version - 7.0.1
ES version - 4.7.4

  1. After patching activity, i have rebooted the Splunk cluster servers, ES search heads, Indexers and clustermaster, deployment server.

  2. After rebooting, ES is not showcasing Notables. and it is not even showcasing results for notables macro.

  3. Error coming as "Error in 'lookup' command: Lookups: The lookup table 'notable_xref_lookup' does not exist or is not available."
    query: $SPLUNK_HOME/bin/splunk search "| rest /services/server/info splunk_server=* | fields splunk_server, kvStoreStatus"

I have checked the clustermaster for kvstore results:
clustermaster failed
deployment server failed
deployeres1 failed
non-es failed
EnterPrise security Search Head Instances are-------- starting
idx 1,2,3,4,5,6,7 ready

Can any one help me here.

0 Karma
1 Solution

shirishkamat84
Path Finder

This was previously mentioned in a different post. Here is what you need to know:

check the expiry of the existing server.pem file, run the below command
$SPLUNK_HOME/bin/splunk cmd openssl x509 -enddate -noout -in /opt/splunk/etc/auth/server.pem

if the certificate is expired, then generate a new one which fixed the issue for us.

Generate the new cert:
mv $SPLUNK_HOME/etc/auth/server.pem $SPLUNK_HOME/etc/auth/server.pem.old

restart Splunk and this should fix the issue. Run this on the instance where KV store is being used.

View solution in original post

shirishkamat84
Path Finder

This was previously mentioned in a different post. Here is what you need to know:

check the expiry of the existing server.pem file, run the below command
$SPLUNK_HOME/bin/splunk cmd openssl x509 -enddate -noout -in /opt/splunk/etc/auth/server.pem

if the certificate is expired, then generate a new one which fixed the issue for us.

Generate the new cert:
mv $SPLUNK_HOME/etc/auth/server.pem $SPLUNK_HOME/etc/auth/server.pem.old

restart Splunk and this should fix the issue. Run this on the instance where KV store is being used.

harsmarvania57
SplunkTrust
SplunkTrust

Hi,

notable macro is depend on notable_xref_lookup and notable_xref_lookup is KVStore lookup so you need to wait till your KVStore will be UP on ES Search Head Instances. If KVStore took too much time start then you need to check $SPLUNK_HOME/var/log/splunk/splunkd.log , $SPLUNK_HOME/var/log/splunk/mongod.log and $SPLUNK_HOME/var/log/introspection/kvstore.log & check why KVStore is taking more time to start.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...