Hey Everyone,
I wanted to see if anyone could help me with correlation searches firing and creating a notable event on the Incident Review page but then not producing the same 1 for 1 match when I run the search manually.
What I did was look at a specific correlation search that fired in the Incident Review page over the last 24 hrs. I then took that search and ran it in a new search with the 24 hr time frame picker. The notable events said that 77 events for that correlation search existed but the search results would return either a 0 or varying numbers if let it finish and ran over and over a few times (none of them being 77). I made sure it wasn't a count issue where an event had multiple counts that in total added up to the total number but was only shown as one row.
The issue seems to be the data models. I run the searches from the index(s) and get vastly different numbers than the Incident Review page which is vastly different than the data model correlation search.
Does anyone have any ideas on why I'm not getting a 1=1=1 match between the Incident Review, correlation search with data models, and the raw index searches?
Any and all help/insight is greatly appreciated!
There's some info about troubleshooting notables...
https://docs.splunk.com/Documentation/ES/6.6.0/Admin/Troubleshootnotables
Let me know if it turns out to be something else.