Splunk Enterprise Security

How to remove custom correlation searches in Splunk Enterprise Security?

kausar
Path Finder

I've been trying to remove some custom correlation searches, but they are still generating notables. So far I've tried:

  • disabling them in savedsearches.conf, but didn't work
  • removing them from the savedsearches.conf and correlationsearches.conf files, still didn't help

What am I missing? I see the names of these correlation rules in ./SA-ThreatIntelligence/lookups/correlationsearchmeta.csv and ./SA-ThreatIntelligence/local/eventtypes.conf files, any idea what they are for and should I delete the entries from here as well? Thanks

hazekamp
Builder

Disabling the saved search configurations for a correlation search will successfully prevent notable events from being generated (since this search will not be dispatched by the scheduler).

Removing the search from savedsearches.conf and correlationsearches.conf will also successfully prevent notable events from being generated (since the search no longer exists).

Perhaps this was done via the configuration file system without refreshing splunkd (this could attribute to the search still being recognized and scheduled by splunkd). Perhaps if the search was a RT search, the currently running search may need to be finalized (this could attribute to the continued generation of notable events).

With respect to correlationsearchmeta.csv, this is a legacy cache that in no way affects the generation of notable events.
With respect to eventtypes.conf, references to the correlation search represent notable event suppressions (filters) of notable events pertaining to the correlation search and in no way affects the generation of notable events.

Hope this helps.

David

kausar
Path Finder

Hi David,

Thanks your reply. This is not something I tried today or yesterday, it has been at least a week since I suppressed all the notables, cleaned/removed the rules from the savedsearches.conf and correlationsearches.conf files and restarted splunk many time after that.

It is very weird that these notables are still being generated (I search for past 24 hours everyday).
I noticed that this is the issue with custom correlation rules of which some were RT but mostly scheduled.

0 Karma

hazekamp
Builder

kauser,

Thanks for the clarifications. Notable events are simply the results of saved searches which persist into the "notable" index. If notable events are still being created then it's highly likely that the search is still running or being executed by the scheduler. I would consult the search audit logs for the saved search in question to determine whether it is in fact still running or being executed. I would also run:

| rest splunk_server=local count=0 /servicesNS/-/-/saved/searches | search title=<search name>

Or consult the saved search manager to ensure that the search is in fact "gone". You are also encouraged to file a support ticket so that we can investigate this more in-depth if needbe.

Thanks,
David

kausar
Path Finder

I don't see deleted saved searches when run the query,

| rest /servicesNS/-/-/saved/searches | search title="*correlation rule*" | table title, search 

OR used btool,

./splunk cmd btool --debug savedsearches list | grep -i "correlation rule"

But I do see them in index=_audit.

Filed a ticket as suggested. Thanks!

0 Karma

hgrow
Communicator

Maybe silly but have you restarted your splunk after you cleaned the savedsearch.conf or did a debug-refresh?

Like @hazekamp mentioned "Perhaps this was done via the configuration file system without refreshing splunkd..."

0 Karma
Get Updates on the Splunk Community!

Optimize Cloud Monitoring

  TECH TALKS Optimize Cloud Monitoring Tuesday, August 13, 2024  |  11:00AM–12:00PM PST   Register to ...

What's New in Splunk Cloud Platform 9.2.2403?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.2.2403! Analysts can ...

Stay Connected: Your Guide to July and August Tech Talks, Office Hours, and Webinars!

Dive into our sizzling summer lineup for July and August Community Office Hours and Tech Talks. Scroll down to ...