Splunk App for Enterprise Security: How to troubleshoot why some custom correlation searches (Notables) are not generating events?

[MHibbin]

Hi,

I've hit a bit of a road block trying to set up some custom correlation searches, which are very similar to others that work successfully.

The data model is configured and generates events; and I have a pivot search that generates events. When setting up others, this worked fine, however, on the last two I've tried setting, it appears to not be generating events. The time frame is the last 60 minutes, and the notable has some variables.

The inspector shows that it is not able to find events (considering the search runs fine in flashtimeline). I know it is a bit ambiguous, but is there anything obvious I could be missing?

hmmm....

Thanks. 🙂

[MHibbin · Accepted solution]

It turns out the correlation searches didn't like the eventtype, I have temporary work around and will have to revisit this at a later date.

[tskinnerivsec]

1st, you need to test the triggering condition, then manually run your search and verify that you are capturing results with the searches that you are using for your correlations.

[jwelch_splunk]

Are they Scheduled or RT Searches? How many cores are on your ES SH? I assume you are created custom DM's for these or are you using one out of the box? Are they accelerated?

[tskinnerivsec]

I would take a serious look at time synchronization across your environment. Correlation searches depend heavily on all the monitored devices having the same time. They should all be using ntp and syncing from the same times source.

[MHibbin]

Scheduled searches,
16 cores,
Custom DM,
Not accelerated 🙂