Splunk Enterprise Security

Enterprise Security: How can I trace the notable events?

danielbb
Motivator

I created a correlation search that should have produced notable events. How can I trace these notable events?

0 Karma
1 Solution

solarboyz1
Builder

When your correlation search runs, it should produce a log also:

index=_* component=SavedSplunker status=success sourcetype=scheduler

08-16-2019 18:08:25.124 +0000 INFO SavedSplunker - savedsearch_id="nobody;DA-ESS-EndpointProtection;Threat -customt - Rule", search_type="", user="me@here", app="DA-ESS-EndpointProtection", savedsearch_name="Threat - custom - Rule", priority=default, status=success, digest_mode=1, scheduled_time=1565978880, window_time=0, dispatch_time=1565978881, run_time=3.585, result_count=0, alert_actions="", sid="scheduler_ZGF2aWQuc2NobWVsdHpAdmEuZ292_REEtRVNTLUVuZHBvaW50UHJvdGVjdGlvbg__RMD528c6568bd07320c3_at_1565978880_0_916EF5DB-94A8-4135-BC30-1E725A10F8A5", suppressed=0, thread_id="AlertNotifierWorker-0", workload_pool=""

You can see if the search produced results, and if any of those results were suppressed.

View solution in original post

solarboyz1
Builder

When your correlation search runs, it should produce a log also:

index=_* component=SavedSplunker status=success sourcetype=scheduler

08-16-2019 18:08:25.124 +0000 INFO SavedSplunker - savedsearch_id="nobody;DA-ESS-EndpointProtection;Threat -customt - Rule", search_type="", user="me@here", app="DA-ESS-EndpointProtection", savedsearch_name="Threat - custom - Rule", priority=default, status=success, digest_mode=1, scheduled_time=1565978880, window_time=0, dispatch_time=1565978881, run_time=3.585, result_count=0, alert_actions="", sid="scheduler_ZGF2aWQuc2NobWVsdHpAdmEuZ292_REEtRVNTLUVuZHBvaW50UHJvdGVjdGlvbg__RMD528c6568bd07320c3_at_1565978880_0_916EF5DB-94A8-4135-BC30-1E725A10F8A5", suppressed=0, thread_id="AlertNotifierWorker-0", workload_pool=""

You can see if the search produced results, and if any of those results were suppressed.

View solution in original post

danielbb
Motivator

@solarboyz1, the notable index does exist ; -)

A follow-up at Enterprise Security: why don't the events get indexed to the notable index?

0 Karma

solarboyz1
Builder

Do you see a SavedSplunker event for the search running?

If results>1 and suppress<1 then the event should show alert_actions="notable"

If it shows notable, then you need to figure out why your search head is not forwarding the notable events to your indexing tier.

If it doesn't show notable, you'll need to troubleshoot that.

0 Karma

danielbb
Motivator

I see - alert_actions is empty for the 10 events I found that match your criteria.

0 Karma

solarboyz1
Builder

Did you add the notable as an adaptive response to the correlation search?

Check the _internal logs for the search id (sid), and check for errors?

0 Karma

danielbb
Motivator

The correlation search has the Adaptive Response action but the Next Steps part is empty now.. not sure if it matters.

index=_internal sid="<sid>"

shows a queued INFO event and the event for the run itself with the dispatch_time and the alert_actions as empty.

Added Next Steps, to be safe, as [[action|nslookup]].

Even added an action - Send email. Maybe this was missing but still alert_actions is empty.

0 Karma

solarboyz1
Builder

There should be no issues with an empty "next steps"

So, after adding the mail as an adaptive response your alert_actions field is still empty. Are you getting emails?

0 Karma

danielbb
Motivator

Right, alert_actions is still empty and I do get the e-mails...

0 Karma

danielbb
Motivator

Based on Creating a notable event from correlation search

Configure > Incident Management > New Notable Event

Now, index=notable shows this single event.

So, what do I miss with the correlation searches set-up?

No worries, I see them ....

0 Karma

solarboyz1
Builder

So, your notable index works and your search head can send notables to it.

Have you tried Creating a notable event using the search language by including | sendalert notable at the end of your search string?

Make sure that the correlation search only outputs the fields you really need and that the fields don't include lots of content (like XML or excessive amounts of text). This can make it difficult for Splunk to parse the stash file. If it cannot parse the stash file, then your notable events may not be generated correctly.

danielbb
Motivator

Much appreciated @solartrek. Notable Event framework in Splunk ES seems to be good.

0 Karma

danielbb
Motivator

So I see in the event suppressed=0 and result_count=1. What does it mean?

0 Karma

solarboyz1
Builder

result_count=1 <- the number of results returned by the correlation search.

suppressed=0 <- This is true (1) or false (0) to indicate if any of the results were suppressed due to throttling.

solarboyz1
Builder

If you see a result_count>0 and a suppressed<1 you should see a notable event for the search (index=notable).

danielbb
Motivator

Interesting, index=notable is empty.

0 Karma

solarboyz1
Builder

Do you have a separate search head(s) and indexers?

Are your search head(s) configured to forward events to your indexers?

Do your indexers have a notable index created?

danielbb
Motivator

Right right. The notable index doesn't exist based on the Cluster Master. with the other two questions we are ok.

What else needs to be created beside this index?

0 Karma

solarboyz1
Builder

If the search head is forwarding events to the indexing tier, and the index exists, the notable should get created.

That said, if you don't have the notable index, you are probable missing other ES specific indexes (https://docs.splunk.com/Documentation/ES/5.3.1/Install/Indexes), which could impact your correlation searches.

You may want to revisit your ES install, https://docs.splunk.com/Documentation/ES/5.3.1/Install/DeploymentPlanning#Using_the_deployment_serve...

danielbb
Motivator

Very kind of you @solarboyz1 - thank you a bunch!!!!

0 Karma

Tune In & Win!

Don't miss out on your
chance to take home free
prizes by helping our players
save the Splunk Cloudom!

Dungeons & Data
Monsters: Splunk O11y
Day Editions Games
stream live:
5/4 at 6:30pm PST
5/5 at 7:00pm PST
on