Splunk Search

Splunk alert for missing logs

chetan1974
Engager

Hi,

Below is a snippet of log pattern generating tons of record. Intending to write a alert if any log are missing for given time time range.
sourcetype source activity
sourcetype1 myLog.log activity1
sourcetype1 myLog.log activity2
sourcetype2 myLog.log activity3
sourcetype2 myLog.log activity3
sourcetype3 myLog.log activity1
sourcetype3 myLog.log activity2
sourcetype3 myLog.log activity3

Is a search, lookup or simple individual query on sourcetype best approach for making sure logs are generated for each sourcetype? Looking for best approach. Thanks.

Tags (1)
0 Karma

gjanders
SplunkTrust
SplunkTrust

MetaWoot from Splunkbase would be a nice solution here, alternatively you can use tstats for example:

| tstats count where sourcetype=sourcetype1 OR sourcetype=sourcetype2 groupby sourcetype, _time span=10m

And then you could find blocks of 10 minutes where the logs are not there or similar

Or perhaps as per my new update something like:

 | tstats max(_indextime) AS mostRecent where sourcetype=sourcetype1 OR sourcetype=sourcetype2 groupby sourcetype
| where mostRecent < now()-600

The above would need to be run over 20 minutes or similar (every 10 minutes) and it advises if a log has not sent any data in for 10 minutes. This may be too sensitive and you will need to adjust it for your environment.

chetan1974
Engager

okay. However, need to get sourcetype names with zero counts. With the query, results includes only sourcetypes with count greater than 0. My intent is to report sourcetypes with missing logs.

0 Karma

gjanders
SplunkTrust
SplunkTrust

The metadata command will do some of what you expect it can tell you when a sourcetype last sent data, but it's not broken down per host.
I'd recommend metawoot as the metadata command isn't as flexible in terms of the data it provides.

An alternative example for tstats would be:

 | tstats max(_indextime) AS mostRecent where sourcetype=sourcetype1 OR sourcetype=sourcetype2 groupby sourcetype
| where mostRecent < now()-600

For example, that would find anything that is not sent in the last 10 minutes, the search can run over the last 20 minutes and it should still be very quick, to make it faster you can add an index=... into the where clause.

I would schedule something that looks back 20 minutes every 10 minutes or similar, or adjust the timing as it's appropriate for your environment.

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...