Missing events in Notable Index Splunk Enterprise Security

[courtneyj]

Hello,

For your awareness my architecture consist of 1SH, 1 Enterprise Security SH,  Cluster of 3 indexes, deployment server with a cluster master, license master, and MC.

 I noticed there are no Notable Events being populated into my notable index. I created events that matched the correlation searches I turned on and also ran those CS searches in search separately to ensure it picked up the events I created. I validated the data models with pivot to ensure data was populating. I also tried to created a manual notable event and nothing showed up in Incident Review. Upon looking at the indexes in the setting menu I see a notable index but nothing is getting populated, likely because I am searching off my index cluster. My deployment server is only managing my core Splunk search head and the I read somewhere that the Splunk_SA_CIM app  needs to have a index.conf  for notable events to be place locally on ES. Can someone please provide some thoughts or suggestions. Thanks in advance..

[lkutch_splunk]

Here are some other tips for troubleshooting notable events: 

https://docs.splunk.com/Documentation/ES/6.4.1/Admin/Troubleshootnotables

[courtneyj]

Yeah I figured out what it was with a little assistance from Splunk Technical support. Since my ES search head was searching off a index cluster the notable index and all other ES required indexes in my index.conf were missing.  Once I added those indexes and applied the cluster bundle it worked like a charm. Also be advised if you go into your cluster master and don't see the indexes show right away that's because the indexes don't show up until something is written to them.

https://docs.splunk.com/Documentation/ES/6.4.1/Install/Indexes

[richgalloway]

Shame on you for letting PS leave without verifying ES is working as it should.  Shame on PS for the same.

---

nothing is getting populated, likely because I am searching off my index cluster

All searches run off the indexer cluster.  If nothing is populated then it's because there's no data or because the search is faulty.

It's helpful for every SH to have a local copy of indexes.conf.  It probably won't solve this problem, but may help with some UI elements.

The notable index is only populated when correlation searches find events that meet their rules AND when the CS has an action to create a notable event.