Splunk Enterprise Security

Why are our notables are not visible in the Splunk Enterprise Security Incident review dashboard?

srisahitya_v
Communicator

Dear Team,

Splunk version - 7.0.1
ES version - 4.7.4

  1. After patching activity, i have rebooted the Splunk cluster servers, ES search heads, Indexers and clustermaster, deployment server.

  2. After rebooting, ES is not showcasing Notables. and it is not even showcasing results for notables macro.

  3. Error coming as "Error in 'lookup' command: Lookups: The lookup table 'notable_xref_lookup' does not exist or is not available."
    query: $SPLUNK_HOME/bin/splunk search "| rest /services/server/info splunk_server=* | fields splunk_server, kvStoreStatus"

I have checked the clustermaster for kvstore results:
clustermaster failed
deployment server failed
deployeres1 failed
non-es failed
EnterPrise security Search Head Instances are-------- starting
idx 1,2,3,4,5,6,7 ready

Can any one help me here.

0 Karma
1 Solution

shirishkamat84
Path Finder

This was previously mentioned in a different post. Here is what you need to know:

check the expiry of the existing server.pem file, run the below command
$SPLUNK_HOME/bin/splunk cmd openssl x509 -enddate -noout -in /opt/splunk/etc/auth/server.pem

if the certificate is expired, then generate a new one which fixed the issue for us.

Generate the new cert:
mv $SPLUNK_HOME/etc/auth/server.pem $SPLUNK_HOME/etc/auth/server.pem.old

restart Splunk and this should fix the issue. Run this on the instance where KV store is being used.

View solution in original post

shirishkamat84
Path Finder

This was previously mentioned in a different post. Here is what you need to know:

check the expiry of the existing server.pem file, run the below command
$SPLUNK_HOME/bin/splunk cmd openssl x509 -enddate -noout -in /opt/splunk/etc/auth/server.pem

if the certificate is expired, then generate a new one which fixed the issue for us.

Generate the new cert:
mv $SPLUNK_HOME/etc/auth/server.pem $SPLUNK_HOME/etc/auth/server.pem.old

restart Splunk and this should fix the issue. Run this on the instance where KV store is being used.

harsmarvania57
SplunkTrust
SplunkTrust

Hi,

notable macro is depend on notable_xref_lookup and notable_xref_lookup is KVStore lookup so you need to wait till your KVStore will be UP on ES Search Head Instances. If KVStore took too much time start then you need to check $SPLUNK_HOME/var/log/splunk/splunkd.log , $SPLUNK_HOME/var/log/splunk/mongod.log and $SPLUNK_HOME/var/log/introspection/kvstore.log & check why KVStore is taking more time to start.

0 Karma
Get Updates on the Splunk Community!

The Splunk Success Framework: Your Guide to Successful Splunk Implementations

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...

Splunk Training for All: Meet Aspiring Cybersecurity Analyst, Marc Alicea

Splunk Education believes in the value of training and certification in today’s rapidly-changing data-driven ...

Investigate Security and Threat Detection with VirusTotal and Splunk Integration

As security threats and their complexities surge, security analysts deal with increased challenges and ...