topic Re: Enterprise Security: How can I trace the notable events? in Splunk Enterprise Security

Enterprise Security: How can I trace the notable events?

danielbb — Fri, 16 Aug 2019 18:05:28 GMT

I created a correlation search that should have produced notable events. How can I trace these notable events?

Re: Enterprise Security: How can I trace the notable events?

solarboyz1 — Wed, 30 Sep 2020 01:49:21 GMT

When your correlation search runs, it should produce a log also:

index=_* component=SavedSplunker status=success sourcetype=scheduler

08-16-2019 18:08:25.124 +0000 INFO SavedSplunker - savedsearch_id="nobody;DA-ESS-EndpointProtection;Threat -customt - Rule", search_type="", user="me@here", app="DA-ESS-EndpointProtection", savedsearch_name="Threat - custom - Rule", priority=default, status=success, digest_mode=1, scheduled_time=1565978880, window_time=0, dispatch_time=1565978881, run_time=3.585, result_count=0, alert_actions="", sid="scheduler_ZGF2aWQuc2NobWVsdHpAdmEuZ292_REEtRVNTLUVuZHBvaW50UHJvdGVjdGlvbg__RMD528c6568bd07320c3_at_1565978880_0_916EF5DB-94A8-4135-BC30-1E725A10F8A5", suppressed=0, thread_id="AlertNotifierWorker-0", workload_pool=""

You can see if the search produced results, and if any of those results were suppressed.

Re: Enterprise Security: How can I trace the notable events?

danielbb — Fri, 16 Aug 2019 18:20:20 GMT

So I see in the event suppressed=0 and result_count=1. What does it mean?

Re: Enterprise Security: How can I trace the notable events?

solarboyz1 — Fri, 16 Aug 2019 18:29:02 GMT

result_count=1 <- the number of results returned by the correlation search.

suppressed=0 <- This is true (1) or false (0) to indicate if any of the results were suppressed due to throttling.

Re: Enterprise Security: How can I trace the notable events?

solarboyz1 — Fri, 16 Aug 2019 18:48:30 GMT

If you see a result_count>0 and a suppressed<1 you should see a notable event for the search (index=notable).

Re: Enterprise Security: How can I trace the notable events?

danielbb — Fri, 16 Aug 2019 18:58:01 GMT

Interesting, index=notable is empty.

Re: Enterprise Security: How can I trace the notable events?

solarboyz1 — Fri, 16 Aug 2019 19:04:24 GMT

Do you have a separate search head(s) and indexers?

Are your search head(s) configured to forward events to your indexers?

Do your indexers have a notable index created?

Re: Enterprise Security: How can I trace the notable events?

danielbb — Fri, 16 Aug 2019 19:15:03 GMT

Right right. The notable index doesn't exist based on the Cluster Master. with the other two questions we are ok.

What else needs to be created beside this index?

Re: Enterprise Security: How can I trace the notable events?

solarboyz1 — Fri, 16 Aug 2019 19:23:49 GMT

If the search head is forwarding events to the indexing tier, and the index exists, the notable should get created.

That said, if you don't have the notable index, you are probable missing other ES specific indexes (https://docs.splunk.com/Documentation/ES/5.3.1/Install/Indexes), which could impact your correlation searches.

You may want to revisit your ES install, https://docs.splunk.com/Documentation/ES/5.3.1/Install/DeploymentPlanning#Using_the_deployment_server_with_Splunk_Enterprise_Security

Re: Enterprise Security: How can I trace the notable events?

danielbb — Fri, 16 Aug 2019 19:42:35 GMT

Very kind of you @solarboyz1 - thank you a bunch!!!!

Re: Enterprise Security: How can I trace the notable events?

danielbb — Thu, 22 Aug 2019 13:46:11 GMT

@solarboyz1, the notable index does exist ; -)

A follow-up at Enterprise Security: why don't the events get indexed to the notable index?

Re: Enterprise Security: How can I trace the notable events?

solarboyz1 — Thu, 22 Aug 2019 15:02:25 GMT

Do you see a SavedSplunker event for the search running?

If results>1 and suppress<1 then the event should show alert_actions="notable"

If it shows notable, then you need to figure out why your search head is not forwarding the notable events to your indexing tier.

If it doesn't show notable, you'll need to troubleshoot that.

Re: Enterprise Security: How can I trace the notable events?

danielbb — Thu, 22 Aug 2019 15:07:09 GMT

I see - alert_actions is empty for the 10 events I found that match your criteria.

Re: Enterprise Security: How can I trace the notable events?

solarboyz1 — Thu, 22 Aug 2019 15:23:53 GMT

Did you add the notable as an adaptive response to the correlation search?

Check the _internal logs for the search id (sid), and check for errors?

Re: Enterprise Security: How can I trace the notable events?

danielbb — Thu, 22 Aug 2019 17:08:14 GMT

The correlation search has the Adaptive Response action but the Next Steps part is empty now.. not sure if it matters.

index=_internal sid="<sid>"

shows a queued INFO event and the event for the run itself with the dispatch_time and the alert_actions as empty.

Added Next Steps, to be safe, as [[action|nslookup]].

Even added an action - Send email. Maybe this was missing but still alert_actions is empty.

Re: Enterprise Security: How can I trace the notable events?

solarboyz1 — Thu, 22 Aug 2019 19:10:19 GMT

There should be no issues with an empty "next steps"

So, after adding the mail as an adaptive response your alert_actions field is still empty. Are you getting emails?

Re: Enterprise Security: How can I trace the notable events?

danielbb — Thu, 22 Aug 2019 19:39:38 GMT

Right, alert_actions is still empty and I do get the e-mails...

Re: Enterprise Security: How can I trace the notable events?

danielbb — Tue, 27 Aug 2019 18:45:58 GMT

Based on Creating a notable event from correlation search

Configure > Incident Management > New Notable Event

Now, index=notable shows this single event.

So, what do I miss with the correlation searches set-up?

No worries, I see them ....

Re: Enterprise Security: How can I trace the notable events?

solarboyz1 — Tue, 27 Aug 2019 19:21:33 GMT

So, your notable index works and your search head can send notables to it.

Have you tried Creating a notable event using the search language by including | sendalert notable at the end of your search string?

Make sure that the correlation search only outputs the fields you really need and that the fields don't include lots of content (like XML or excessive amounts of text). This can make it difficult for Splunk to parse the stash file. If it cannot parse the stash file, then your notable events may not be generated correctly.

Re: Enterprise Security: How can I trace the notable events?

danielbb — Tue, 03 Sep 2019 15:29:06 GMT

Much appreciated @solartrek. Notable Event framework in Splunk ES seems to be good.