Monitoring log sources that goes silent?

[richa]

Hi.

I'm looking for a query/solution that will alert me when a log source is no longer sending logs. We have 4 indexes to monitor with a lot of log sources. So, having the log sources in input lookup would not be a good idea as it would have to be maintained every time new log source is added. Thus, i am looking for a query which alerts me if any of the log sources currently configured in any of the 4 indexes goes silent for 24 hours. Would prefer not to have lookup command in the query as file would have to be maintained in that scenario. Need to run this query on all the currently configured log sources.

Thank you.

[PickleRick]

There are several ready-made apps aiming at that.

You can also try to look on your index data "backwards" and compare with - let's say - last day if you don't want to have a predefined set of sources/sourcetypes/hosts/whatever that you will look for but that approach is prone to timing out your alerts - if you search 3 days backwards, it won't alert you of the sources that used to send data 4 days ago but stopped after that.

So you can only do so much. There are no miracles and Splunk doesn't know what "is supposed" to be sent to it unless you tell it.

[gcusello]

Hi @richa,

having a lookup will give you more conterol on the check process, but anyway, if you want to check the sourcetypes in four indexes that didn't send logs in the last 24 hours but that sent in the previous 30 days, you can run the following search:

| tstats 
   count 
   latest(_time) AS _time 
   values(index) AS index 
WHERE 
   index IN (index1, index2, index3, index4) 
   earliest=-30d@d 
BY sourcetype
| eval period=if(_time>now()-86400,"Last 24 hours", "Previous")
| stats 
   dc(period) AS period_count 
   values(period) AS period 
   BY sourcetype
| where period_count=1 AND period="Previous"
| table sourcetype index _time

Ciao.

Giuseppe

[richa]

Hi @gcusello,

Thanks for the query. But I would like to have it on sources and not source type as certain source types can have few log sources configured in it. Also, one more thing i would like to add, there are few log files which are created daily based on date. And if we configure the alert on log sources then those log sources will come as alert in the search query for not receiving logs as daily new files are generated. so, how to cater to that scenario?

[gcusello]

Hi @richa ,

probably, source isn't the best solution to check ingestion because it usually change, for this reason I used sourcetype, otherwise, you could use index:

| tstats 
   count 
   latest(_time) AS _time 
WHERE 
   index IN (index1, index2, index3, index4) 
   earliest=-30d@d 
BY index
| eval period=if(_time>now()-86400,"Last 24 hours", "Previous")
| stats 
   dc(period) AS period_count 
   values(period) AS period 
   BY index
| where period_count=1 AND period="Previous"
| table index _time

Ciao.

Giuseppe

[scelikok]

Hi @richa,

You can use the below query by updating your 4 indexes, it will check all sources that send data in 48 hours but stopped more than 24 hours.

| tstats latest(_indextime) as latest where index IN (index1,index2,index3,index4) earliest=-48h by source 
| eval delay = now() -latest 
| where delay > 86400 
| eval delay=tostring(delay, "duration") 
| fields - latest

[richa]

Hi @scelikok,

Thanks for the query. This query is not giving any info about the index. Can index field be added as well while displaying the results. Also,  there are certain source log files which are created daily based on date. So, how will that be taken into account?

[scelikok]

You can see the index of the source by using below query;

| tstats latest(_indextime) as latest where index IN (index1,index2,index3,index4) earliest=-48h by source index
| eval delay = now() -latest 
| where delay > 86400 
| eval delay=tostring(delay, "duration") 
| fields - latest

Since above query check the latest 48 hours ingested events. Filters the results that do not send data for at least 24 hours. Looking for 48 hours back will make sure daily updated sources will taken into account.

[richa]

Hi, I checked the solution. Seems like there are still issues with this solution. As it will not show event logs generated 24 hours back but it is showing daily events which were generated 48 hours back. Eg: if i check it today, it will not show logs sources generated on 26-feb but it will show logs sources generated on 25 feb date. 

[richa]

Hi, Can you help?

[scelikok]

Hi @richa,

Since you asked for alerting data sources that stopped for more than 24 hours, it will not show yesterday's logs.

You can change the delay parameter according to your needs. 

86400 is equivalent to 24 hours in seconds.