How do I determine why a Correlation Search isn't creating a notable event when I expected one?

[LukeMurphey]

I have a Correlation Search that didn't generate notable events in a couple where I think it should have. How can I determine why it didn't work?

[stroud_bc]

Another problem that can cause notables from being created is if a lookup definition (transform) created by the user and referenced by the correlation search has Sharing configured to Private. This results in a search that runs fine for you, but simply returns zero results when run by the scheduler, with no evidence of skips, errors, or suppression in the logs.

Setting the Sharing setting to Global allows the correlation search to run normally. This can be done under Settings -> Lookups -> Lookup Definitions.

I imagine the same problem would arise if the CSV or KV store itself is permissions-restricted, but the above is the problem I encountered.

[mmontgomery_spl]

Also, Splunk 7.2 allows you to not use the "admin" user during setup. I have found ES relies on this user for some processes, so you should still use the name "admin" in the setup process of installing Splunk on an ES search head.

[mosman_splunk]

Plus
make sure that the saved searches belong to existing user, otherwise it will be orphan searches will will not generate notable.

[LukeMurphey · Accepted solution]

Here are a few things to check:

1) Run the search manually over the given time frame and see if it matches the events.
If it doesn't match, remove parts of the search until you isolate the part of the search that doesn't match.

2) Check the search scheduler logs.
Run the following search to check the search scheduler logs:

index=_internal sourcetype=scheduler

Here are some things to look for:

• Make sure that the search ran during the time-frame that you expected events
• See if suppressed is indicates if events were suppressed
• See if result_count indicates notable events were created (i.e. is greater than one)
• Check the status field to make sure that the search ran successfully.

3) Check to see if the notable index contains the notable events.
Do a search against the notable index directly and determine if the notable event exists but is being excluded for some reason:

index=notable

Note that suppressions will filter notable events from appearing on Incident Review. If you see your notable event here, then make sure that no suppressions are preventing the notable event from appearing on Incident Review.

4) Check the notable alert action logs.

These logs will indicate whether the notable alert action was triggered to make a notable event. Below is a search to view these logs:

index=_internal sourcetype=notable_modalert

5) Make sure the search output doesn't include lots of extraneous output.

Make sure that the correlation search only outputs the fields you really need and that the fields don't include lots of content (like XML or excessive amounts of text). This can make it difficult for Splunk to parse the stash file. If it cannot parse the stash file, then your notable events may not be generated correctly.