My goal for this search is to find if a file was not imported. If the file is imported "Could not find a file in the" text will be present. If the file is imported, "Moved" text will be present. Our system looks to import the file consistantly. So, at 07:30:00 the file may be imported so "Moved" will appear in the log file. But at 08:00:00 the system will look to import the file, but since there is nothing to import (because it was imported at 07:30:00) "Could not file the file in the" will be written to the log. I am trying to write a search that if "Moved" is present then I do not even want to know the count of "Could not find the file in the". However, if "Moved" is never found in the log file during the specified time frame, then I do want to run a search and find all instances of "Could not find the file in the". So in short, I am seeking help of writing a search that does not include false negatives.
Below is the search I have put together. I attemped to search for the expected files and find a count. Then in the second query, if the count was less than expected (so I did not find the number of imported files as expected), then I would look for "Could not find the file in the", the files that were not found. This search is not working as I have found I can not pass a count's value into a sub-search/second query. So "FilesImported" from the first query does not have the value I expect in the second query's "WHERE" clause. I have shortened the query for just two instances of a customers (instead of all 20+) as each customer has different files that are to be imported at different times on different days.
source=*D:\\redacted\\redacted* source=*IH_Daily\\redacted* Moved earliest=-48h@h
| eval dayBuffer=strftime(now(), "%d") | eval day=ltrim(tostring(dayBuffer),"0")
| eval todayBuffer=strftime(now(), "%m_"+day+"_%Y") | eval today=ltrim(tostring(todayBuffer),"0") | where like(source,"%10_19_2017%")
| eval time=strftime(round(strptime(file_Time, "%I:%M:%S %P")), "%H:%M:%S")
| eval dow=strftime(strptime(file_Date, "%m/%d/%Y"), "%A")
| rex field=source "redacted\\\+(?<ClientID>[^\\\]+)"
| where ClientID="$clientID$"
| where (like(source,"%"."client1"."%") AND (dow!="Sunday" AND dow!="Monday") AND (time>"07:27:00" AND time<"08:27:00") AND (FileImported="file1")
OR (like(source,"%"."client2"."%")) AND (FilesImported!=2) AND (dow!="Sunday" AND dow!="Tuesday") AND (time>"09:00:00"AND time<"11:30:00") AND ((FileImported="file1") OR (FileImported="file2") OR (FileImported="file3"))))
| stats count as FilesImported
| append [ search source=*D:\\redacted\\redacted* source=*IH_Daily\\redacted* ("Could not find a file in the") earliest=-48h@h
| eval dayBuffer=strftime(now(), "%d") | eval day=ltrim(tostring(dayBuffer),"0")
| eval todayBuffer=strftime(now(), "%m_"+day+"_%Y") | eval today=ltrim(tostring(todayBuffer),"0") | where like(source,"%10_19_2017%")
| eval time=strftime(round(strptime(file_Time, "%I:%M:%S %P")), "%H:%M:%S")
| eval dow=strftime(strptime(file_Date, "%m/%d/%Y"), "%A")
| rex field=source "redacted\\\+(?<ClientID>[^\\\]+)"
| where ClientID="$clientID$"
| where ((like(source,"%"."client1"."%")) AND (FilesImported<1) AND (dow!="Sunday" AND dow!="Monday") AND (time>"07:27:00"AND time<"08:27:00") AND (file_Missing="file1")
OR (like(source,"%"."client2"."%")) AND (FilesImported!<3) AND (dow!="Sunday" AND dow!="Tuesday") AND (time>"09:00:00"AND time<"11:30:00") AND ((file_Missing="file1") OR (file_Missing="file2") OR (file_Missing="file3")))
| stats count as "File Missed" ]
|table "File Missed"
... View more