topic Re: Splunk alert for missing logs in Splunk Search

Splunk alert for missing logs

chetan1974 — Thu, 05 Oct 2017 20:12:40 GMT

Hi,

Below is a snippet of log pattern generating tons of record. Intending to write a alert if any log are missing for given time time range.
sourcetype source activity
sourcetype1 myLog.log activity1
sourcetype1 myLog.log activity2
sourcetype2 myLog.log activity3
sourcetype2 myLog.log activity3
sourcetype3 myLog.log activity1
sourcetype3 myLog.log activity2
sourcetype3 myLog.log activity3

Is a search, lookup or simple individual query on sourcetype best approach for making sure logs are generated for each sourcetype? Looking for best approach. Thanks.

Re: Splunk alert for missing logs

gjanders — Fri, 06 Oct 2017 02:16:51 GMT

MetaWoot from Splunkbase would be a nice solution here, alternatively you can use tstats for example:

| tstats count where sourcetype=sourcetype1 OR sourcetype=sourcetype2 groupby sourcetype, _time span=10m

And then you could find blocks of 10 minutes where the logs are not there or similar

Or perhaps as per my new update something like:

 | tstats max(_indextime) AS mostRecent where sourcetype=sourcetype1 OR sourcetype=sourcetype2 groupby sourcetype
| where mostRecent < now()-600

The above would need to be run over 20 minutes or similar (every 10 minutes) and it advises if a log has not sent any data in for 10 minutes. This may be too sensitive and you will need to adjust it for your environment.

Re: Splunk alert for missing logs

chetan1974 — Fri, 06 Oct 2017 19:13:03 GMT

okay. However, need to get sourcetype names with zero counts. With the query, results includes only sourcetypes with count greater than 0. My intent is to report sourcetypes with missing logs.

Re: Splunk alert for missing logs

gjanders — Fri, 06 Oct 2017 21:24:01 GMT

The metadata command will do some of what you expect it can tell you when a sourcetype last sent data, but it's not broken down per host.
I'd recommend metawoot as the metadata command isn't as flexible in terms of the data it provides.

An alternative example for tstats would be:

 | tstats max(_indextime) AS mostRecent where sourcetype=sourcetype1 OR sourcetype=sourcetype2 groupby sourcetype
| where mostRecent < now()-600

For example, that would find anything that is not sent in the last 10 minutes, the search can run over the last 20 minutes and it should still be very quick, to make it faster you can add an index=... into the where clause.

I would schedule something that looks back 20 minutes every 10 minutes or similar, or adjust the timing as it's appropriate for your environment.