How to best do deal with alerting and summary indexing on mobile laptops

[jambajuice]

If I want to use Splunk to monitor event logs on laptops that will be on and offline with some frequency, how does Splunk deal with alerting and summary indexing on data that comes in much later than the original event? Will an event that contains a much older timestamp trigger a realtime alert when the indexer sees it for the first time?

What about periodic summary indexing? If I run a summary indexing search every hour and Splunk receives events from a laptop with timestamps that are 24 hours old, will those events show up in the search for events from the last hour?

Thx.

Craig

[hexx]

One thing that you could do if you want to run a real-time search that looks at all incoming events regardless of their time-stamp (and alerts on them) is to use the "real-time(all time)" time range. For a scheduled RT search, this means that both the lower and upper time range of your search would have to be set to "rt". Be careful to properly set up your alert conditions accordingly so as not create false positives!

I would recommend that you first give a try to this time range from the search app to see what incoming events look like. Just search for "*" with the "real-time(all time)" range selected from the field picker to see what's coming in.

[dwaddle]

I suspect you'll be in for a rough time of it. Remember, both basic alerting (not realtime alerting) and summary indexing are based upon running scheduled searches over a time range. Existing best practice is to give forwarded data some "lag time" before running a summary index or alert search -- for example, on hourly summaries run at 5 minutes past the hour with a timerange of earliest=-65m@m latest=-5m@m. This gives you 5 minutes of lag time for events to arrive and make it into the index before the summary / alert search runs.

I think that what will happen that your summaries will miss data and any basic alerts you set up won't fire -- because the data won't appear in the index until much, much later than the scheduled search runs.

I don't know enough about real-time alerts to be able to comment on how they will handle this.