Splunk Search

How to search for and trigger an email alert when there are no logs generated in a directory?

venkatdba64
New Member

I am needing to create an Alert to run every 30 minutes to monitor the file size of all the log files in a directory /opt/data/cassandra/logs. When the logs directory is empty, I have to set up an alert to trigger an email. How to write a Splunk search for this? Thanks

I have /opt/data/cassandra/logs

Whenever there are no logs generated in this log directory, I have to send an alert email.

0 Karma

MikeElliott
Communicator

Hi there,

Maybe this logic will be of assistance?

This logic is designed to create an alert for investigation where Splunk has detected no events within a given sourcetype within the past 2 hours. It does not address your concern of log files not being created, but would be a good indicator that this has not happened.

| tstats count dc(host) AS distinct_hosts latest(_time) AS latest_time WHERE index=* BY sourcetype
| where latest_time<=relative_time(now(), "-2h") 
| sort -latest_time 
| convert timeformat="%H:%M:%S %d/%m/%Y" ctime(latest_time)
| where count=0
0 Karma

woodcock
Esteemed Legend

Are you using batch with move_policy = sinkhole and forwarding the logs into Splunk?
Have you configured your own scripted input to do /bin/ls -altr /opt/data/cassandra/logs/ | wc -l | awk '{$1 = $1 - 3; print $1}'?
What EXACTLY are you pushing into splunk that would allow you to see if the directory is empty or not?

0 Karma

frobinson_splun
Splunk Employee
Splunk Employee

Hi @venkatdba64,
It seems like you might be able to use an advanced conditional alert, where you apply additional custom conditions to your search results in order to manage alert triggering. For example, if your search returns no log files, or some other indicator that the directory is empty (perhaps total file size == 0?), then you can trigger the alert.

See this topic in our documentation:
http://docs.splunk.com/Documentation/Splunk/6.3.0/Alert/Definescheduledalerts#Set_up_triggering_cond...

I'm not sure what version of the product you're using, but when you set up an alert, you can set up trigger conditions for it. You can create a custom condition expression to evaluate the search results you get. This custom condition helps manage alert triggering.

I hope this helps! Let me know if not.

Best,
@frobinson_splunk

0 Karma

venkatdba64
New Member

I am using splunk 6.2 . can I monitor the disk space of hosts from splunk gui like . I am good to set the alert but my problem to how to get the info with query

I am trying this
index=aws-** host=venkatdba64 | timechart avg("%_Free_Space") by host . but this is not working for me

host freespace usedspace

0 Karma

steveyz
Splunk Employee
Splunk Employee

Splunk doesn't monitor the hosts directly, but you can use the unix app, or simply configure a scripted input to run a shell command like df periodically to report the free disk space.

frobinson_splun
Splunk Employee
Splunk Employee

Ok. I'm not sure about your query, where you're searching for "host=venkatdba64" and then piping the result to your timechart command and trying to average by host. Are there other hosts whose percentage of free space you are trying to capture?

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...