Splunk Search

How do you build an alert that triggers when a file is moved to a monitored folder?

kozanic_FF
Path Finder

I'm trying to build an alert that triggers when a file is moved to an Error folder within the system we are monitoring.

There are a few exceptions that I've needed to factor into my search some easy to figure out - others more difficult - one in particular.

My current Search:

index=* sourcetype="FLO_LOG_FILES" DirPath=*\\Error* NOT 
 [| inputlookup ErrorFileExclude-Thresholds
  | eval path = if(len(FileName)>2,DirPath.FileName,"")
  | fields path]
| lookup IndexToClient index output ClientName Environment
| lookup ErrorFileExclude-Thresholds ClientName DirPath output FolderTimeTH FileCountTH
| fillnull value=0 FileCountTH
| fillnull value=1440 FolderTimeTH
| eval MinsDetected = round((now()-_time)/60,0)
| search MinsDetected > FolderTimeTH
| table host ClientName Environment source DirPath FolderTimeTH MinsDetected FileCountTH

The first NOT excludes particular file names - this part works well.

The Bottom search should be excluding files based on an time threshold for that files folder. This is where I'm having issues.

If I update either side of the comparison with an actual number e.g. MinsDetected > 1440 OR FolderTimeTH < 1440 the filter works as expected, yet when I have the variable on each side - it's not working - struggling to understand why not.

Based on the above search the last line in the below results should be the only result - yet it is not:

alt text

Anyone able to provide assistance on this issue?

0 Karma
1 Solution

kozanic_FF
Path Finder

Managed to figure out a way to get what I needed as mentioned above - full solution for anyone interested below:

index=* sourcetype="FLO_LOG_FILES" DirPath=*\\Error* NOT 
 [| inputlookup ErrorFileExclude-Thresholds
  | fillnull value=0 FileName_DayTmp
  | fillnull value=0 FileName
  | eval FileName=if(len(FileName_DayTmp)>1,FileName_DayTmp,FileName)
  | eval path = if(len(FileName)>1,DirPath.FileName,"")
  | fields path]
| lookup IndexToClient index output ClientName Environment
| lookup ErrorFileExclude-Thresholds ClientName DirPath output FolderTimeTH_DayTmp FolderTimeTH FileCountTH_DayTmp FileCountTH
| fillnull value=0 FileCountTH
| fillnull value=1440 FolderTimeTH
| fillnull value=0 FileCountTH_DayTmp
| fillnull value=0 FolderTimeTH_DayTmp
| eval FileCountTH=if(FileCountTH_DayTmp!=0,FileCountTH_DayTmp,FileCountTH)
| eval FolderTimeTH=if(FolderTimeTH_DayTmp!=0,FolderTimeTH_DayTmp,FolderTimeTH)
| eval MinsDetected = round((now()-_time)/60,0)
| eval AlertCheck = if(MinsDetected<FolderTimeTH,"True","False")
| search AlertCheck = True
| stats count as NumFiles by index host ClientName Environment source DirPath FileCountTH
| where NumFiles > FileCountTH

This search allows me to monitor a set of folders named "ERROR" and alert based on the below:

  • File name is not on the exclusion list
  • Time since detected for the Folder instance is less than the threshold
  • Total count of files is less than the threshold for the given folder instance
  • There is an option on both Time and Count thresholds to have fixed and DayTemp Threshold - lookup file is refreshed daily to clear the temp values

View solution in original post

0 Karma

kozanic_FF
Path Finder

Managed to figure out a way to get what I needed as mentioned above - full solution for anyone interested below:

index=* sourcetype="FLO_LOG_FILES" DirPath=*\\Error* NOT 
 [| inputlookup ErrorFileExclude-Thresholds
  | fillnull value=0 FileName_DayTmp
  | fillnull value=0 FileName
  | eval FileName=if(len(FileName_DayTmp)>1,FileName_DayTmp,FileName)
  | eval path = if(len(FileName)>1,DirPath.FileName,"")
  | fields path]
| lookup IndexToClient index output ClientName Environment
| lookup ErrorFileExclude-Thresholds ClientName DirPath output FolderTimeTH_DayTmp FolderTimeTH FileCountTH_DayTmp FileCountTH
| fillnull value=0 FileCountTH
| fillnull value=1440 FolderTimeTH
| fillnull value=0 FileCountTH_DayTmp
| fillnull value=0 FolderTimeTH_DayTmp
| eval FileCountTH=if(FileCountTH_DayTmp!=0,FileCountTH_DayTmp,FileCountTH)
| eval FolderTimeTH=if(FolderTimeTH_DayTmp!=0,FolderTimeTH_DayTmp,FolderTimeTH)
| eval MinsDetected = round((now()-_time)/60,0)
| eval AlertCheck = if(MinsDetected<FolderTimeTH,"True","False")
| search AlertCheck = True
| stats count as NumFiles by index host ClientName Environment source DirPath FileCountTH
| where NumFiles > FileCountTH

This search allows me to monitor a set of folders named "ERROR" and alert based on the below:

  • File name is not on the exclusion list
  • Time since detected for the Folder instance is less than the threshold
  • Total count of files is less than the threshold for the given folder instance
  • There is an option on both Time and Count thresholds to have fixed and DayTemp Threshold - lookup file is refreshed daily to clear the temp values
0 Karma

richgalloway
SplunkTrust
SplunkTrust

@kozanic_FF If your problem is resolved, please accept an answer to help future readers.

---
If this reply helps you, Karma would be appreciated.
0 Karma

kozanic_FF
Path Finder

Hi RIch,

I just had to wait until my answer was posted before I could set as accepted answer - I don't have enough karma points yet for my posts to appear straight away 😞

0 Karma

richgalloway
SplunkTrust
SplunkTrust

That's the nature of search - it doesn't support fields compared to fields. Try where, instead. It should to the job.

---
If this reply helps you, Karma would be appreciated.

richgalloway
SplunkTrust
SplunkTrust

You can have more than one where clause.

---
If this reply helps you, Karma would be appreciated.
0 Karma

kozanic_FF
Path Finder

Understand that - but was struggling to get the result I was after

0 Karma

kozanic_FF
Path Finder

Thanks for the response Rich, unfortunately I need to use the where clause for another filter.
I have tried using where to combine both this other filter with the one I'm having issues with - but getting similar results.

I have come up with a work around however:

| eval AlertCheck = if(MinsDetected<FolderTimeTH,"True","False")
| search AlertCheck = True
0 Karma
Get Updates on the Splunk Community!

Optimize Cloud Monitoring

  TECH TALKS Optimize Cloud Monitoring Tuesday, August 13, 2024  |  11:00AM–12:00PM PST   Register to ...

What's New in Splunk Cloud Platform 9.2.2403?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.2.2403! Analysts can ...

Stay Connected: Your Guide to July and August Tech Talks, Office Hours, and Webinars!

Dive into our sizzling summer lineup for July and August Community Office Hours and Tech Talks. Scroll down to ...