How can I send a notification when file is missing?

[appsik]

Hello dear community,

I am new here and hope for warm support.

The following problem I have to solve: I have several files and if a document is missing, should be sent a notification with the reference to this file.

Example:
File12324.txt
File21111.txt
Filefdfdf.txt
(naming without pattern)
If next day File21111.txt is missing, email goes out with content "..." + File21111.txt + "..."

Thanks for the advice

[gcusello]

Hi @appsik,

you should put the filenames to check in a lookup (called e.g. filenames.csv) containing at least a field called "filename".

Then you should run a simple search like the following:

index=your_index
| stats count BY filename
| append [ | inputlookup filename.csv | eval count=0 | fields filename count ]
| stats sum(count) AS total BY filename
| where total=0

If you have problems to extract the filename field from your logs, I can help you but you should share some sample of your logs.

Ciao.

Giuseppe

[appsik]

Hi @gcusello 

Many thanks for your help!

Method with lookup table:
I have created and uploaded a table:

I have assigned FileA.csv, FileB.csv, FileC.csv to my_index

"FileD.csv" is missing in my_index and should be send by email.

What am I doing wrong?

[gcusello]

Hi @appsik ,

there's a difference in the filename field extracted from the index and the one in the lookup.

run the search without the last row and see if there are differences.

Please, share results and code in text mode (using the "Insert/Edit Code sample" button) instead as a screenshot, so I can use it.

Ciao.

Giuseppe

[appsik]

@gcusello unfortunately I do not see any differences

index=my_index
| stats count BY Filename
| append [ | inputlookup Lookup_table_with_filename1.csv | eval count=0 | fields Filename count ]
| stats sum(count) AS Total BY Filename

[gcusello]

Hi @appsik,

also uppercase and lowercase?

could you share the results of the search without the last row?

Ciao.

Giuseppe

[appsik]

@gcusello 

After run this:

index=my_index
| stats count BY Filename
| append [ | inputlookup Lookup_table_with_filename1.csv | eval count=0 | fields Filename count ]
| stats sum(count) AS Total BY Filename

I see:

[gcusello]

Hi @appsik ,

which results if you run only the first two rows?

index=my_index
| stats count BY Filename

If you haven't any result the problem is the main search.

Ciao.

Giuseppe

[appsik]

@gcusello 

I see: 3 events

but no results

Thank you for your time!

[gcusello · Accepted solution]

Hi @appsik ,

this means that you have three events but yu haven't the field Filename.

If you run only the main search (first row) in Verbose Mode, have you this field in Interesting fields?

probably not, so try to run this:

index=my_index
| rex field=source "(?<Filename>\w+\.txt)$"
| stats count BY Filename
| append [ | inputlookup Lookup_table_with_filename1.csv | eval count=0 | fields Filename count ]
| stats sum(count) AS Total BY Filename

Ciao.

Giuseppe

[appsik]

@gcusello 

>this means that you have three events but yu haven't the field Filename.

>If you run only the main search (first row) in Verbose Mode, have you this field in Interesting fields?

>probably not

Thank you, I understood that

Now please return to my task

if I run:

index=my_index
| rex field=source "(?<Filename>\w+\.csv)$"
| stats count BY Filename
| append [ | inputlookup Lookup_table_with_filename1.csv | eval count=0 | fields Filename count ]
| stats sum(count) AS Total BY Filename

I see:

I don't understand what this (second line) line is for. | rex field=source "(?<Filename>\w+\.csv)$"
Is it possible without RegEx?

[gcusello]

Hi @appsik,

in the second raw I extract the filename from the source using a regex.

Anyway, regex is the only way to extract a parte or the source field.

If you don't know very well regexes I hint to use some time to learn this because regexes are very usegul in Splunk.

Ciao.

Giuseppe

[appsik]

@gcusello 

I have the problem not with RegEx, but with the variable: Filename. Where is this variable initialized

[gcusello]

Hi @appsik,

Filename isn't a variable but a field extracted using the regex from the source field in the second row of the search.

Ciao.

Giuseppe

[appsik]

@gcusello 

rex field=source "(?<Filename>\w+\.csv)$"
? -> Zero or None
<Filename> -> Could you please explain what exactly <Filename> is causing here
w+ -> One or more letter, number, or _...
.csv)$ Name of file must end with .csv

[appsik]

@gcusello ok, I'll try to understand it later.

Now I have to do my task.
When I have this output:

Can I create an alert like this:

Trigger condition: search Total != 1
Is there a more elegant solution?

[gcusello]

Hi @appsik,

yes insert again the last row we deleted for troubleshooting (| where total=0) and use Results>0 as condition for your alert.

Ciao.

Giuseppe

[appsik]

Hi @gcusello 

>(| where total=0)
yes, that makes sense
>Results>0
yes, that makes sense too

please stay available, now i try without lookup table. Thank you very much for the support

[gcusello]

Hi @appsik ,

be sure I'm always here!

if one answer solves your need, please accept one answer for the other people of Community or tell me how I can help you.

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated 😉

[appsik]

Hi @gcusello 

Thanks for the quick reply!
I am not allowed to create a new file. It is only 2-3 files. Later I can catch it with regex

What I think:

index=o sourcetyp=txt source="*\file1.txt"
OR "*\file2.txt"
OR "*\file3.txt" // if possible
| stats sum(source) AS total BY fileName - stats dc(source) as fileCount // if file2.txt is missing
| where total != 0

-> alert trigger, I need name of "file2.txt" to send by email

Sorry, I am absolute beginner here

[gcusello]

Hi @appsik,

if you cannot creat a lookup (check this because it's strange!) and you have to check only few files, please try this:

index=your_index (source=*\file1.txt OR source=*\file2.txt OR source=*\file3.txt)
| rex field=source "(?<filename>\w+\.txt)$"
| stats count BY filename
| append [ | makeresults | eval filename=file1.txt, count=0 | fields filename count ]
| append [ | makeresults | eval filename=file2.txt, count=0 | fields filename count ]
| append [ | makeresults | eval filename=file3.txt, count=0 | fields filename count ]
| stats sum(count) AS total BY filename
| where total=0

Ciao.

Giuseppe