We had an incident on a device that we had not had a chance to ingest logs into Splunk. That incident occurred 2 weeks ago. That device is now feeding logs into Splunk. It ingested the logs on that device that were up to a year old. The problem is, the timeframe that the incident happened, did not trigger any of the ESS out-of-the-box Correlation Searches\Notable events (because the timestamps were outside of the search criteria of the Correlation search). I went to the device and downloaded the logs from the time we started ingesting them into Splunk going back 90 days. I also setup a test index to ingest those logs into Splunk.
Issues I am having:
1) How do I get the Correlation Searches to look at data that is 90+ days old?
2) Since a bunch of the out-of-the-box Correlation Searches in ESS utilize the data models, how does that affect how these trigger the alerts with a timestamp that is that far out?
Basically, I am looking to see if any of the Correlation Searches\Notable events that come with ESS would have triggered and alerted on the incident we had. This will tell me whether I need to tune those alerts, or whether I need to create custom Correlation Searches to detect this incident in the future.
Most likely you'll need to do a custom search to look into whatever the CS was looking into. You can see that some CS's run in scheduled times looking back (e.g. run every 15min looking into the last hour (like a rolling window)) and you can't really backfill these.
Regarding the datamodels it will depend on what is the time interval you set for each data model acceleration. if the data model is accelerating say 1 year worth of data, it will eventually pick up the events you indexed from 90 days ago. So even with indexing these dated events you can still benefit from building a search based on the CS you were looking at and targeting the datamodel instead of the raw data.
If you need help building a search for this case based on some CS, post here which CS you are looking at.
Well, that is one of the things I am stuck on. These logs are CIM compliant and feed into multiple data models. So there are a handful of CS's that could trigger on these logs. I was wondering if there is a way to just have all correlation searches run against the older data, or do I have to go in and customize every CS to look back at that time frame (and just the ones dealing with those data models). Also, if some alerts do grouping (i.e. by src_ip), will these trigger one alert for the whole 90+ days, or individual alerts as it would on a CS that has an hour scheduled run time?
if you know details regarding the incident you might as well start with the CSs that would be triggered in that case. That might narrow down the number of CS to "customise". Regarding grouping it will most likely depend on how will your custom search be like. you could use streamstats with some attributes (window, time_window, reset_after, etc ) to simulate the rolling window effect of the scheduled searches and group this by your grouping field.