Hi,
Below is a snippet of log pattern generating tons of record. Intending to write a alert if any log are missing for given time time range.
sourcetype source activity
sourcetype1 myLog.log activity1
sourcetype1 myLog.log activity2
sourcetype2 myLog.log activity3
sourcetype2 myLog.log activity3
sourcetype3 myLog.log activity1
sourcetype3 myLog.log activity2
sourcetype3 myLog.log activity3
Is a search, lookup or simple individual query on sourcetype best approach for making sure logs are generated for each sourcetype? Looking for best approach. Thanks.
MetaWoot from Splunkbase would be a nice solution here, alternatively you can use tstats for example:
| tstats count where sourcetype=sourcetype1 OR sourcetype=sourcetype2 groupby sourcetype, _time span=10m
And then you could find blocks of 10 minutes where the logs are not there or similar
Or perhaps as per my new update something like:
| tstats max(_indextime) AS mostRecent where sourcetype=sourcetype1 OR sourcetype=sourcetype2 groupby sourcetype
| where mostRecent < now()-600
The above would need to be run over 20 minutes or similar (every 10 minutes) and it advises if a log has not sent any data in for 10 minutes. This may be too sensitive and you will need to adjust it for your environment.
okay. However, need to get sourcetype names with zero counts. With the query, results includes only sourcetypes with count greater than 0. My intent is to report sourcetypes with missing logs.
The metadata command will do some of what you expect it can tell you when a sourcetype last sent data, but it's not broken down per host.
I'd recommend metawoot as the metadata command isn't as flexible in terms of the data it provides.
An alternative example for tstats would be:
| tstats max(_indextime) AS mostRecent where sourcetype=sourcetype1 OR sourcetype=sourcetype2 groupby sourcetype
| where mostRecent < now()-600
For example, that would find anything that is not sent in the last 10 minutes, the search can run over the last 20 minutes and it should still be very quick, to make it faster you can add an index=... into the where clause.
I would schedule something that looks back 20 minutes every 10 minutes or similar, or adjust the timing as it's appropriate for your environment.