How to create a triggering email alert to include log files content in the email?

[rchittip]

Hi,
I would like to set up email alerts for the log content of files. If file stuck in the directory, would like to send alert with filename to the team. Can we achieve this in splunk. Below are the logs.

[2018/02/05 07:26][Warning]:[3PL] : File EDIDATA7129 is stuck in queue for more than 4 hrs. FileName: **D:\EDI\3PL\Inbound\EDIDATA7129**
[2018/02/05 07:26][Warning]:[3PL] : File EDIDATA6077 is stuck in queue for more than 4 hrs. FileName: **D:\EDI\3PL\Inbound\EDIDATA6077**
[2018/02/05 07:26][Warning]:[3PL] : File EDIDATA6074 is stuck in queue for more than 4 hrs. FileName: **D:\EDI\3PL\Inbound\EDIDATA6074**

Thanks,
Ramu Chittiprolu

[jkat54 · Accepted solution]

Yes, this is possible. Start by extracting the file name:

index=yourindex "stuck in queue" | rex "FileName:\s(?<file>.+)" 

Then create a list of file names and date/time/host

index=yourindex "stuck in queue" | rex "FileName:\s(?<file>.+)"  | table _time host file

now save this as an alert, for the trigger action add "send email", select your options on how to attach the data... schedule the alert to run however often you like.

To avoid issues with bad timestamps, use _index_earliest in your alert (optional):

index=yourindex _index_earliest=-15m  "stuck in queue" | rex "FileName:\s(?<file>.+)" | table _time host file

The search above when scheduled to run every 15 minutes, would find any events indexed in the last 15 minutes even if timestamp recognition is off and regardless of indexing latency.

I edited to remove _index_latest=@m. When you snap to on the latest and you search doesnt execute at the minute you could miss the events that come between when your search actually executes and the last time. best just to leave that off the search.