How to create an alert if any one log file is missing from long list of files from same location?

[abhi04]

We need to create an alert if any one log file is missing from long list of files from same location.

index= index=auto_prod_cm_comparisions source=*cert_details.log |stats values(source) by host

[cpetterborg · Accepted solution]

You can also use some multi search trickery. This isn't a solution to your problem, but is an example of what you can use to see the differences between two different time periods. Let's start with this example:

| multisearch 
    [search index=main earliest=-2h@h latest=-1h@h
    |  eval type="foo" ] 
    [search index=main earliest=-1h@h latest=-0h@h
    |  eval type="bar"] 
| stats count by host, type 
| fields - count
| stats count by host
| search count=1

This is just taking the list of hosts from one hour and comparing to the list of hosts from another hour. The list is then counted by host names, and if there are less than two occurrences of a given host from the list, then it shows as a count of 1 for that host, which is to be considered a single occurrence from the two time periods. This could be an extra one from either the first list or second, which list is not determined from this search, only that it exists in one, but not the other. This method doesn't require keeping a master list, or a summary search entry for the previous search.

You would certainly have to modify this type of search to match your particular case. Without having some of your data to work with it isn't easy coming up with a complete solution.

[abhi04]

Hi @cpetterborg,

I tried with lookup table. In lookup table I have mentioned all the three host and 279 sources.
I used the below query but I am only getting three sources which are there in lookup but not in the splunk events but should be more than that.

i am using the below query:

|inputlookup source.csv | search NOT [ search index=auto_prod_cm_comparisions source=*cert_details.log earliest=-1m ] | dedup host,source|fields host,source

[jowenssi]

NOT and dedup are really expensive. Try something like this:

search index=auto_prod_cm_comparisions source=*cert_details.log earliest=-1m | stats count by host,source | fields host,source | append [inputlookup source.csv | stats count by host,source | fields host,source] | stats count by host, source | search count = 1

That will give you the delta between the two and allow you to identify which sources are either extra or missing.

[abhi04]

The log files have name starting with the time stamp. so the filename changes everyday.
e.g.

062118_0300_CAP_dfw1svpap504_cert_details.log
062118_0300_CAP_dfw1svpap505_cert_details.log
062118_0300_PROD_dfw1svpap507_cert_details.log
062118_0300_PROD_sat1svmap536_cert_details.log
062118_0300_PROD_sat1svmap538_cert_details.log
062018_0200_CAP_dfw1svpap504_cert_details.log
062018_0300_CAP_dfw1svpap505_cert_details.log
062018_0250_PROD_dfw1svpap507_cert_details.log
062018_0310_PROD_sat1svmap536_cert_details.log
062018_0300_PROD_sat1svmap538_cert_details.log
where "062018_0300" is date and time stamp.
So I have taken only the portion "PROD_sat1svmap538_cert_details.log" for comparing. But then at last I have to add the date and time to the portion to display full file name for current day.

[jowenssi]

There are a couple ways to do this. You could keep a "master list" of log files as a CSV lookup, then have a scheduled search compare your search against the lookup and send an alert if there are less records.

The other way would be to index (or append to a KVstore) the number/list of log files for each host and do the same scheduled search comparison.

Basically, you'll need to decide if you want to compare against a master list, or compare against a previous ("normal") list of log files.

[abhi04]

And I have more than 50 log files per host