I created a correlation search that should have produced notable events. How can I trace these notable events?
When your correlation search runs, it should produce a log also:
index=_* component=SavedSplunker status=success sourcetype=scheduler
08-16-2019 18:08:25.124 +0000 INFO SavedSplunker - savedsearch_id="nobody;DA-ESS-EndpointProtection;Threat -customt - Rule", search_type="", user="me@here", app="DA-ESS-EndpointProtection", savedsearch_name="Threat - custom - Rule", priority=default, status=success, digest_mode=1, scheduled_time=1565978880, window_time=0, dispatch_time=1565978881, run_time=3.585, result_count=0, alert_actions="", sid="scheduler_ZGF2aWQuc2NobWVsdHpAdmEuZ292_REEtRVNTLUVuZHBvaW50UHJvdGVjdGlvbg__RMD528c6568bd07320c3_at_1565978880_0_916EF5DB-94A8-4135-BC30-1E725A10F8A5", suppressed=0, thread_id="AlertNotifierWorker-0", workload_pool=""
You can see if the search produced results, and if any of those results were suppressed.
When your correlation search runs, it should produce a log also:
index=_* component=SavedSplunker status=success sourcetype=scheduler
08-16-2019 18:08:25.124 +0000 INFO SavedSplunker - savedsearch_id="nobody;DA-ESS-EndpointProtection;Threat -customt - Rule", search_type="", user="me@here", app="DA-ESS-EndpointProtection", savedsearch_name="Threat - custom - Rule", priority=default, status=success, digest_mode=1, scheduled_time=1565978880, window_time=0, dispatch_time=1565978881, run_time=3.585, result_count=0, alert_actions="", sid="scheduler_ZGF2aWQuc2NobWVsdHpAdmEuZ292_REEtRVNTLUVuZHBvaW50UHJvdGVjdGlvbg__RMD528c6568bd07320c3_at_1565978880_0_916EF5DB-94A8-4135-BC30-1E725A10F8A5", suppressed=0, thread_id="AlertNotifierWorker-0", workload_pool=""
You can see if the search produced results, and if any of those results were suppressed.
@solarboyz1, the notable index does exist ; -)
A follow-up at Enterprise Security: why don't the events get indexed to the notable index?
Do you see a SavedSplunker event for the search running?
If results>1 and suppress<1 then the event should show alert_actions="notable"
If it shows notable, then you need to figure out why your search head is not forwarding the notable events to your indexing tier.
If it doesn't show notable, you'll need to troubleshoot that.
I see - alert_actions
is empty for the 10 events I found that match your criteria.
Did you add the notable as an adaptive response to the correlation search?
Check the _internal logs for the search id (sid), and check for errors?
The correlation search has the Adaptive Response action but the Next Steps part is empty now.. not sure if it matters.
index=_internal sid="<sid>"
shows a queued INFO event and the event for the run itself with the dispatch_time
and the alert_actions
as empty.
Added Next Steps, to be safe, as [[action|nslookup]]
.
Even added an action - Send email. Maybe this was missing but still alert_actions
is empty.
There should be no issues with an empty "next steps"
So, after adding the mail as an adaptive response your alert_actions field is still empty. Are you getting emails?
Right, alert_actions
is still empty and I do get the e-mails...
Based on Creating a notable event from correlation search
Configure > Incident Management > New Notable Event
Now, index=notable
shows this single event.
So, what do I miss with the correlation searches set-up?
No worries, I see them ....
So, your notable index works and your search head can send notables to it.
Have you tried Creating a notable event using the search language by including | sendalert notable at the end of your search string?
Make sure that the correlation search only outputs the fields you really need and that the fields don't include lots of content (like XML or excessive amounts of text). This can make it difficult for Splunk to parse the stash file. If it cannot parse the stash file, then your notable events may not be generated correctly.
Much appreciated @solartrek. Notable Event framework in Splunk ES seems to be good.
So I see in the event suppressed=0
and result_count=1
. What does it mean?
result_count=1 <- the number of results returned by the correlation search.
suppressed=0 <- This is true (1) or false (0) to indicate if any of the results were suppressed due to throttling.
If you see a result_count>0 and a suppressed<1 you should see a notable event for the search (index=notable).
Interesting, index=notable
is empty.
Do you have a separate search head(s) and indexers?
Are your search head(s) configured to forward events to your indexers?
Do your indexers have a notable index created?
Right right. The notable index doesn't exist based on the Cluster Master. with the other two questions we are ok.
What else needs to be created beside this index?
If the search head is forwarding events to the indexing tier, and the index exists, the notable should get created.
That said, if you don't have the notable index, you are probable missing other ES specific indexes (https://docs.splunk.com/Documentation/ES/5.3.1/Install/Indexes), which could impact your correlation searches.
You may want to revisit your ES install, https://docs.splunk.com/Documentation/ES/5.3.1/Install/DeploymentPlanning#Using_the_deployment_serve...
Very kind of you @solarboyz1 - thank you a bunch!!!!