Splunk Enterprise Security

Why are our notables are not visible in the Splunk Enterprise Security Incident review dashboard?

srisahitya_v
Communicator

Dear Team,

Splunk version - 7.0.1
ES version - 4.7.4

  1. After patching activity, i have rebooted the Splunk cluster servers, ES search heads, Indexers and clustermaster, deployment server.

  2. After rebooting, ES is not showcasing Notables. and it is not even showcasing results for notables macro.

  3. Error coming as "Error in 'lookup' command: Lookups: The lookup table 'notable_xref_lookup' does not exist or is not available."
    query: $SPLUNK_HOME/bin/splunk search "| rest /services/server/info splunk_server=* | fields splunk_server, kvStoreStatus"

I have checked the clustermaster for kvstore results:
clustermaster failed
deployment server failed
deployeres1 failed
non-es failed
EnterPrise security Search Head Instances are-------- starting
idx 1,2,3,4,5,6,7 ready

Can any one help me here.

0 Karma
1 Solution

shirishkamat84
Path Finder

This was previously mentioned in a different post. Here is what you need to know:

check the expiry of the existing server.pem file, run the below command
$SPLUNK_HOME/bin/splunk cmd openssl x509 -enddate -noout -in /opt/splunk/etc/auth/server.pem

if the certificate is expired, then generate a new one which fixed the issue for us.

Generate the new cert:
mv $SPLUNK_HOME/etc/auth/server.pem $SPLUNK_HOME/etc/auth/server.pem.old

restart Splunk and this should fix the issue. Run this on the instance where KV store is being used.

View solution in original post

shirishkamat84
Path Finder

This was previously mentioned in a different post. Here is what you need to know:

check the expiry of the existing server.pem file, run the below command
$SPLUNK_HOME/bin/splunk cmd openssl x509 -enddate -noout -in /opt/splunk/etc/auth/server.pem

if the certificate is expired, then generate a new one which fixed the issue for us.

Generate the new cert:
mv $SPLUNK_HOME/etc/auth/server.pem $SPLUNK_HOME/etc/auth/server.pem.old

restart Splunk and this should fix the issue. Run this on the instance where KV store is being used.

harsmarvania57
Ultra Champion

Hi,

notable macro is depend on notable_xref_lookup and notable_xref_lookup is KVStore lookup so you need to wait till your KVStore will be UP on ES Search Head Instances. If KVStore took too much time start then you need to check $SPLUNK_HOME/var/log/splunk/splunkd.log , $SPLUNK_HOME/var/log/splunk/mongod.log and $SPLUNK_HOME/var/log/introspection/kvstore.log & check why KVStore is taking more time to start.

0 Karma
Get Updates on the Splunk Community!

Splunk Certification Support Alert | Pearson VUE Outage

Splunk Certification holders and candidates!  Please be advised of an upcoming system maintenance period for ...

Enterprise Security Content Update (ESCU) | New Releases

In September, the Splunk Threat Research Team had two releases of new security content via the Enterprise ...

New in Observability - Improvements to Custom Metrics SLOs, Log Observer Connect & ...

The latest enhancements to the Splunk observability portfolio deliver improved SLO management accuracy, better ...