topic Re: How to search for and trigger an email alert when there are no logs generated in a directory? in Splunk Search

How to search for and trigger an email alert when there are no logs generated in a directory?

venkatdba64 — Tue, 27 Oct 2015 17:07:43 GMT

I am needing to create an Alert to run every 30 minutes to monitor the file size of all the log files in a directory /opt/data/cassandra/logs. When the logs directory is empty, I have to set up an alert to trigger an email. How to write a Splunk search for this? Thanks

I have /opt/data/cassandra/logs

Whenever there are no logs generated in this log directory, I have to send an alert email.

Re: How to search for and trigger an email alert when there are no logs generated in a directory?

frobinson_splun — Tue, 27 Oct 2015 17:33:16 GMT

Hi @venkatdba64,
It seems like you might be able to use an advanced conditional alert, where you apply additional custom conditions to your search results in order to manage alert triggering. For example, if your search returns no log files, or some other indicator that the directory is empty (perhaps total file size == 0?), then you can trigger the alert.

See this topic in our documentation:
http://docs.splunk.com/Documentation/Splunk/6.3.0/Alert/Definescheduledalerts#Set_up_triggering_conditions_for_a_scheduled_alert

I'm not sure what version of the product you're using, but when you set up an alert, you can set up trigger conditions for it. You can create a custom condition expression to evaluate the search results you get. This custom condition helps manage alert triggering.

I hope this helps! Let me know if not.

Best,
@frobinson_splunk

Re: How to search for and trigger an email alert when there are no logs generated in a directory?

venkatdba64 — Tue, 29 Sep 2020 07:41:00 GMT

I am using splunk 6.2 . can I monitor the disk space of hosts from splunk gui like . I am good to set the alert but my problem to how to get the info with query

I am trying this
index=aws-** host=venkatdba64 | timechart avg("%_Free_Space") by host . but this is not working for me

host freespace usedspace

Re: How to search for and trigger an email alert when there are no logs generated in a directory?

frobinson_splun — Tue, 27 Oct 2015 19:01:17 GMT

Ok. I'm not sure about your query, where you're searching for "host=venkatdba64" and then piping the result to your timechart command and trying to average by host. Are there other hosts whose percentage of free space you are trying to capture?

Re: How to search for and trigger an email alert when there are no logs generated in a directory?

steveyz — Tue, 27 Oct 2015 22:30:22 GMT

Splunk doesn't monitor the hosts directly, but you can use the unix app, or simply configure a scripted input to run a shell command like df periodically to report the free disk space.

Re: How to search for and trigger an email alert when there are no logs generated in a directory?

woodcock — Fri, 06 Nov 2015 04:06:48 GMT

Are you using batch with move_policy = sinkhole and forwarding the logs into Splunk?
Have you configured your own scripted input to do /bin/ls -altr /opt/data/cassandra/logs/ | wc -l | awk '{$1 = $1 - 3; print $1}'?
What EXACTLY are you pushing into splunk that would allow you to see if the directory is empty or not?

Re: How to search for and trigger an email alert when there are no logs generated in a directory?

MikeElliott — Wed, 14 Nov 2018 02:49:35 GMT

Hi there,

Maybe this logic will be of assistance?

This logic is designed to create an alert for investigation where Splunk has detected no events within a given sourcetype within the past 2 hours. It does not address your concern of log files not being created, but would be a good indicator that this has not happened.

| tstats count dc(host) AS distinct_hosts latest(_time) AS latest_time WHERE index=* BY sourcetype
| where latest_time<=relative_time(now(), "-2h") 
| sort -latest_time 
| convert timeformat="%H:%M:%S %d/%m/%Y" ctime(latest_time)
| where count=0