topic Re: Alert if no log messages in Alerting

Alert if no log messages

rmacurak — Fri, 01 Oct 2010 21:20:28 GMT

I'm indexing logfiles from a custom web application that receives nonstop traffic, resulting in virtually nonstop log messages. Occasionally (unfortunately) the application will hang, resulting in a gap in the logging. I'd like to be alerted if this condition occurs. Any ideas on how to search/alert in realtime for "gaps" in the log file, or an absence of data?

Re: Alert if no log messages

christopherutz — Fri, 01 Oct 2010 21:23:33 GMT

Our solution for this was to run a saved search every n minutes that searches back n minutes for everything. If the results count is less than 1 we send an email. Although not real time, we use n = 5 which fits our needs.

Re: Alert if no log messages

southeringtonp — Fri, 01 Oct 2010 21:58:52 GMT

If you just want the immediate alert (no historical reporting), you could do something like:

| metadata type=sources | search source=yoursource | eval age=now()-lastTime | search age>60

If it returns any results, your source has not reported in a longer-than-expected amount of time (60 seconds in this case).

Note: It's not clear to me whether it's preferable to use lastTime or RecentTime in the eval statement. Maybe someone else can explain the difference.

Re: Alert if no log messages

rmacurak — Sat, 02 Oct 2010 00:40:17 GMT

Elegant. Thank you very much.

Re: Alert if no log messages

rmacurak — Sat, 02 Oct 2010 00:47:25 GMT

This was my initial idea as well, but I figured there was a better way. This is a good way to look back at historical data, though, which also comes in handy

Re: Alert if no log messages

araitz — Sat, 02 Oct 2010 03:27:37 GMT

lastTime is the time value of the last event received's timestamp.

recentTime is the last time (on the index server) that an event was received.

Re: Alert if no log messages

rmacurak — Mon, 04 Oct 2010 22:59:31 GMT

Unfortunately I was too quick to mark this as solved. It seems that the "age" always grows to 30s and resets to 0s, indicating that the remote Splunk servers that feed this index are only sending data every 30 seconds, even though the logs are constantly written -- does that sound plausible? If so, how do I instruct the Splunk forwarders to stream data in realtime rather than batching every 30 seconds?

Re: Alert if no log messages

southeringtonp — Tue, 05 Oct 2010 00:07:48 GMT

My guess (and only a guess) is that events are streaming, but the metadata only gets updated every 30 seconds. Is that really a problem though? How often do you want to run the saved search?

Re: Alert if no log messages

rmacurak — Tue, 05 Oct 2010 00:17:39 GMT

Perhaps not. Ideally I'd be able to identify 5-10 second gaps in the logs, which indicates the "hanging" condition. I'll investigate your hypothesis, thanks.

Re: Alert if no log messages

bshields1 — Wed, 30 Jan 2013 16:00:23 GMT

So how does the search change to test for a count of at least 1 record? I'm new to Splunk and not sure what to add to the search noted above.