Splunk Enterprise Security

Missing events in Notable Index Splunk Enterprise Security

courtneyj
Engager

Hello,

For your awareness my architecture consist of 1SH, 1 Enterprise Security SH,  Cluster of 3 indexes, deployment server with a cluster master, license master, and MC.

 I noticed there are no Notable Events being populated into my notable index. I created events that matched the correlation searches I turned on and also ran those CS searches in search separately to ensure it picked up the events I created. I validated the data models with pivot to ensure data was populating. I also tried to created a manual notable event and nothing showed up in Incident Review. Upon looking at the indexes in the setting menu I see a notable index but nothing is getting populated, likely because I am searching off my index cluster. My deployment server is only managing my core Splunk search head and the I read somewhere that the Splunk_SA_CIM app  needs to have a index.conf  for notable events to be place locally on ES. Can someone please provide some thoughts or suggestions. Thanks in advance..

0 Karma

lkutch_splunk
Splunk Employee
Splunk Employee

Here are some other tips for troubleshooting notable events: 

https://docs.splunk.com/Documentation/ES/6.4.1/Admin/Troubleshootnotables

courtneyj
Engager

Yeah I figured out what it was with a little assistance from Splunk Technical support. Since my ES search head was searching off a index cluster the notable index and all other ES required indexes in my index.conf were missing.  Once I added those indexes and applied the cluster bundle it worked like a charm. Also be advised if you go into your cluster master and don't see the indexes show right away that's because the indexes don't show up until something is written to them.

 

https://docs.splunk.com/Documentation/ES/6.4.1/Install/Indexes

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Shame on you for letting PS leave without verifying ES is working as it should.  Shame on PS for the same.


nothing is getting populated, likely because I am searching off my index cluster

All searches run off the indexer cluster.  If nothing is populated then it's because there's no data or because the search is faulty.

It's helpful for every SH to have a local copy of indexes.conf.  It probably won't solve this problem, but may help with some UI elements.

The notable index is only populated when correlation searches find events that meet their rules AND when the CS has an action to create a notable event.

---
If this reply helps you, Karma would be appreciated.
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...