Splunk Enterprise Security

Enterprise Security: How can I trace the notable events?

Motivator

I created a correlation search that should have produced notable events. How can I trace these notable events?

0 Karma
1 Solution

Builder

When your correlation search runs, it should produce a log also:

index=_* component=SavedSplunker status=success sourcetype=scheduler

08-16-2019 18:08:25.124 +0000 INFO SavedSplunker - savedsearch_id="nobody;DA-ESS-EndpointProtection;Threat -customt - Rule", search_type="", user="me@here", app="DA-ESS-EndpointProtection", savedsearch_name="Threat - custom - Rule", priority=default, status=success, digest_mode=1, scheduled_time=1565978880, window_time=0, dispatch_time=1565978881, run_time=3.585, result_count=0, alert_actions="", sid="scheduler_ZGF2aWQuc2NobWVsdHpAdmEuZ292_REEtRVNTLUVuZHBvaW50UHJvdGVjdGlvbg__RMD528c6568bd07320c3_at_1565978880_0_916EF5DB-94A8-4135-BC30-1E725A10F8A5", suppressed=0, thread_id="AlertNotifierWorker-0", workload_pool=""

You can see if the search produced results, and if any of those results were suppressed.

View solution in original post

Builder

When your correlation search runs, it should produce a log also:

index=_* component=SavedSplunker status=success sourcetype=scheduler

08-16-2019 18:08:25.124 +0000 INFO SavedSplunker - savedsearch_id="nobody;DA-ESS-EndpointProtection;Threat -customt - Rule", search_type="", user="me@here", app="DA-ESS-EndpointProtection", savedsearch_name="Threat - custom - Rule", priority=default, status=success, digest_mode=1, scheduled_time=1565978880, window_time=0, dispatch_time=1565978881, run_time=3.585, result_count=0, alert_actions="", sid="scheduler_ZGF2aWQuc2NobWVsdHpAdmEuZ292_REEtRVNTLUVuZHBvaW50UHJvdGVjdGlvbg__RMD528c6568bd07320c3_at_1565978880_0_916EF5DB-94A8-4135-BC30-1E725A10F8A5", suppressed=0, thread_id="AlertNotifierWorker-0", workload_pool=""

You can see if the search produced results, and if any of those results were suppressed.

View solution in original post

Motivator

@solarboyz1, the notable index does exist ; -)

A follow-up at Enterprise Security: why don't the events get indexed to the notable index?

0 Karma

Builder

Do you see a SavedSplunker event for the search running?

If results>1 and suppress<1 then the event should show alert_actions="notable"

If it shows notable, then you need to figure out why your search head is not forwarding the notable events to your indexing tier.

If it doesn't show notable, you'll need to troubleshoot that.

0 Karma

Motivator

I see - alert_actions is empty for the 10 events I found that match your criteria.

0 Karma

Builder

Did you add the notable as an adaptive response to the correlation search?

Check the _internal logs for the search id (sid), and check for errors?

0 Karma

Motivator

The correlation search has the Adaptive Response action but the Next Steps part is empty now.. not sure if it matters.

index=_internal sid="<sid>"

shows a queued INFO event and the event for the run itself with the dispatch_time and the alert_actions as empty.

Added Next Steps, to be safe, as [[action|nslookup]].

Even added an action - Send email. Maybe this was missing but still alert_actions is empty.

0 Karma

Builder

There should be no issues with an empty "next steps"

So, after adding the mail as an adaptive response your alert_actions field is still empty. Are you getting emails?

0 Karma

Motivator

Right, alert_actions is still empty and I do get the e-mails...

0 Karma

Motivator

Based on Creating a notable event from correlation search

Configure > Incident Management > New Notable Event

Now, index=notable shows this single event.

So, what do I miss with the correlation searches set-up?

No worries, I see them ....

0 Karma

Builder

So, your notable index works and your search head can send notables to it.

Have you tried Creating a notable event using the search language by including | sendalert notable at the end of your search string?

Make sure that the correlation search only outputs the fields you really need and that the fields don't include lots of content (like XML or excessive amounts of text). This can make it difficult for Splunk to parse the stash file. If it cannot parse the stash file, then your notable events may not be generated correctly.

Motivator

Much appreciated @solartrek. Notable Event framework in Splunk ES seems to be good.

0 Karma

Motivator

So I see in the event suppressed=0 and result_count=1. What does it mean?

0 Karma

Builder

result_count=1 <- the number of results returned by the correlation search.

suppressed=0 <- This is true (1) or false (0) to indicate if any of the results were suppressed due to throttling.

Builder

If you see a result_count>0 and a suppressed<1 you should see a notable event for the search (index=notable).

Motivator

Interesting, index=notable is empty.

0 Karma

Builder

Do you have a separate search head(s) and indexers?

Are your search head(s) configured to forward events to your indexers?

Do your indexers have a notable index created?

Motivator

Right right. The notable index doesn't exist based on the Cluster Master. with the other two questions we are ok.

What else needs to be created beside this index?

0 Karma

Builder

If the search head is forwarding events to the indexing tier, and the index exists, the notable should get created.

That said, if you don't have the notable index, you are probable missing other ES specific indexes (https://docs.splunk.com/Documentation/ES/5.3.1/Install/Indexes), which could impact your correlation searches.

You may want to revisit your ES install, https://docs.splunk.com/Documentation/ES/5.3.1/Install/DeploymentPlanning#Using_the_deployment_serve...

Motivator

Very kind of you @solarboyz1 - thank you a bunch!!!!

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!