Why after operating normally for a 7 days are all my notable events showing "0" for the last 24 hours

[bucknerj]

Splunk PS setup our instance and the last day here the Notable Events began falling. No changes that I am aware of but we have been creating additional data inputs/indexes. I have checked our ingestion and the same events continue to populate the indexes. ES just does not seem to process any. We have over 100 correlation searches that show enabled.

[bucknerj]

Splunk support solved.
Tenable app had loaded a conf that sent notable events to the stash bypassing the notable index. Disabled bad conf lines and it is back up

[woodcock]

You have GOT to be kidding me! Please give us more details. There are several Tenable apps on splunk base; which one (please give URL) and which version? Which lines did you comment out. Please add this detail to this answer then click Accept to close it so we can all avoid/fix this problem in our environments!

[woodcock]

I saved this answer for last because I can't see how it could actually happen. Each Correlation Search in ES starts out with an Alert Action called Notable and this is what generates the notable event in the summary index. Do your Saved Searches still have this Alert Action? Also, did you check this search:

index=_* (WARN OR ERROR) notable*

[bucknerj]

After eliminating search related instances this is the only event occuring:

08-05-2019 07:42:01.526 -0400 ERROR DatabaseDirectoryManager - Failed to remove summary of bid=notable~2~324931C1-6BA3-4CAF-A0E1-B88B4457E0BE with cid="dma|notable~2~324931C1-6BA3-4CAF-A0E1-B88B4457E0BE|FE9350B2-8BA7-4622-886B-62F14F94F944_DM_Splunk_SA_CIM_Authentication" from summary manager, skipping remove.

[bucknerj]

I put the last comment i the answer posts, here is the savedsearches.conf:

[Event Sequencing Engine - Main]
action.keyindicator.invert = 0
action.makestreams.param.verbose = 0
action.nbtstat.param.verbose = 0
action.notable.param.verbose = 0
action.nslookup.param.verbose = 0
action.ping.param.verbose = 0
action.risk.param.verbose = 0
action.threat_add.param.verbose = 0
alert.track = 0
enableSched = 1

[bucknerj]

My last comment did not post:

No event in the summary index - all time

index=_* (WARN OR ERROR) notable* #-- results: No warn, error=0 on all events, hit=0, miss=0
Values Count % #--removed count and % in results
splunkd_remote_searches
audittrail
splunkd
contentinfo_rest_handler
scheduler
splunkd_access
splunk_disk_objects
configuration_check
splunk_resource_usage

[Event Sequencing Engine - Main]
action.keyindicator.invert = 0
action.makestreams.param.verbose = 0
action.nbtstat.param.verbose = 0
action.notable.param.verbose = 0
action.nslookup.param.verbose = 0
action.ping.param.verbose = 0
action.risk.param.verbose = 0
action.threat_add.param.verbose = 0
alert.track = 0
enableSched = 1

[bucknerj]

Many entries, no WARN, error=0, hit=0, miss=0 in each result

savedsearches.conf
[Event Sequencing Engine - Main]
action.keyindicator.invert = 0
action.makestreams.param.verbose = 0
action.nbtstat.param.verbose = 0
action.notable.param.verbose = 0
action.nslookup.param.verbose = 0
action.ping.param.verbose = 0
action.risk.param.verbose = 0
action.threat_add.param.verbose = 0
alert.track = 0
enableSched = 1

[woodcock]

If you are a brand new ES environment and you already have 100 correlation searches enabled then you are DEFINITELY not doing it properly/well. You probably also have a bunch of them set to real-time which craters performance (yes, I know that it is not real real-time but even so, it is bad). The first thing that we do when setting Splunk and/or ES is to disable all real-time permanently. In ES we onboard searches ONE AT A TIME and almost every one of them will require modification for it work well (well). It looks like you got shoddy PS and they just turned everything on and left. That is a recipe for a terrible long-term ES experience.

My mantra with es is: ES is a toolbox, not an appliance: and even then most of the tools in the box are rusty and ill-suited for general use.

[bucknerj]

Disabled all correlation searches, none are real time, validated received events. It took a week to get up to 100 correlations by testing about 10 a day, some returned little some a lot. I have now enabled 26 correlations for about 2 hours and still no notables in Security Posture. I have also validated all indexes are owned by the splunk process running the service.

[lakshman239]

If you run the correlation searches manually on the search (within ES context) for, say last 4 hours, are you getting any results?

[bucknerj]

I have tested 4 manual correlation searches and do get results from the ES Search bar
I ran the Untriaged Notable Events Correlation Search but got no results, I am assuming because the notables are not showing up.
Up to 10 successful manual Correlation searches which are enabled.

[lakshman239]

If running the searches manually works for the time window and if they same don't work via a scheduled search, then either they are skipped are they are running in the context which doesn't have required previleges or there could be other reasons. Suggest raise a support case with splunk, if you rule out permissions/skipped searches/roles etc..

[FrankVl]

Could also be that the data comes in with a delay (possibly caused by onboarding additional data sources, and overwhelming the ingest pipeline at some point), causing the correlation search to not see it, but running the same search later (over a longer window), manually, shows the results.

[woodcock]

Other possible reasons include:

Your DMAs have been turned off.
Your data sources (forwarders) are not sending the raw events.
Your Correlation Searches were turned off (or otherwise broken through modification).

[lakshman239]

Do you really need those 100 correlation searches? If so, I assume the boxes are spec'ed for concurrent searches.

if you run |notable`` for all time, are you seeing any entries? if so, when was the last time you had entries. You can then work out whats happening to the correlation searches since they worked last time.

[woodcock]

There are dozens of reasons that this can happen and many PS houses (including mine) do a brisk business in doing Health Checks on ES and Splunk Core to resolve/detect/prevent them. Here are some things to check.

Do you have skipped searches (run DMC Health Checks)?  You probably are 90%-100% which means notables are not being created.
Does your notable summary index exist and is it OK?
Does the account that you are using have the roles/permissions that are required?

[bucknerj]

1. My ES shows a skip ratio of 0.80%
2. Notable index is healthy
3. I am using an account with Splunk Admin and ess_admin permissions

The server and indexes pass health checks.
The Security Posture dashboard displays "0"s but I do see notables in some of the Security Domains options.