Splunk Search

How can I send a notification when file is missing?

appsik
Explorer

Hello dear community,

I am new here and hope for warm support.

The following problem I have to solve: I have several files and if a document is missing, should be sent a notification with the reference to this file.

Example:
File12324.txt
File21111.txt
Filefdfdf.txt
(naming without pattern)
If next day File21111.txt is missing, email goes out with content "..." + File21111.txt + "..."

Thanks for the advice

Labels (1)
Tags (1)
0 Karma
1 Solution

gcusello
SplunkTrust
SplunkTrust

Hi @appsik ,

this means that you have three events but yu haven't the field Filename.

If you run only the main search (first row) in Verbose Mode, have you this field in Interesting fields?

probably not, so try to run this:

index=my_index
| rex field=source "(?<Filename>\w+\.txt)$"
| stats count BY Filename
| append [ | inputlookup Lookup_table_with_filename1.csv | eval count=0 | fields Filename count ]
| stats sum(count) AS Total BY Filename

Ciao.

Giuseppe

 

View solution in original post

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @appsik,

you should put the filenames to check in a lookup (called e.g. filenames.csv) containing at least a field called "filename".

Then you should run a simple search like the following:

index=your_index
| stats count BY filename
| append [ | inputlookup filename.csv | eval count=0 | fields filename count ]
| stats sum(count) AS total BY filename
| where total=0

If you have problems to extract the filename field from your logs, I can help you but you should share some sample of your logs.

Ciao.

Giuseppe

0 Karma

appsik
Explorer

Hi @gcusello 

Many thanks for your help!

Method with lookup table:
I have created and uploaded a table:
my_lookupTbl.PNG
I have assigned FileA.csv, FileB.csv, FileC.csv to my_index
my_index.PNG
"FileD.csv" is missing in my_index and should be send by email.

my_search_with_lookup.PNG

What am I doing wrong?

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @appsik ,

there's a difference in the filename field extracted from the index and the one in the lookup.

run the search without the last row and see if there are differences.

Please, share results and code in text mode (using the "Insert/Edit Code sample" button) instead as a screenshot, so I can use it.

Ciao.

Giuseppe

0 Karma

appsik
Explorer

@gcusello unfortunately I do not see any differences

index=my_index
| stats count BY Filename
| append [ | inputlookup Lookup_table_with_filename1.csv | eval count=0 | fields Filename count ]
| stats sum(count) AS Total BY Filename

 

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @appsik,

also uppercase and lowercase?

could you share the results of the search without the last row?

Ciao.

Giuseppe

0 Karma

appsik
Explorer

@gcusello 

After run this:

index=my_index
| stats count BY Filename
| append [ | inputlookup Lookup_table_with_filename1.csv | eval count=0 | fields Filename count ]
| stats sum(count) AS Total BY Filename

I see:

my_search1.PNG

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @appsik ,

which results if you run only the first two rows?

index=my_index
| stats count BY Filename

If you haven't any result the problem is the main search.

Ciao.

Giuseppe

0 Karma

appsik
Explorer

@gcusello 

I see: 3 events

but no results

my_search2.PNG

Thank you for your time!

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @appsik ,

this means that you have three events but yu haven't the field Filename.

If you run only the main search (first row) in Verbose Mode, have you this field in Interesting fields?

probably not, so try to run this:

index=my_index
| rex field=source "(?<Filename>\w+\.txt)$"
| stats count BY Filename
| append [ | inputlookup Lookup_table_with_filename1.csv | eval count=0 | fields Filename count ]
| stats sum(count) AS Total BY Filename

Ciao.

Giuseppe

 

0 Karma

appsik
Explorer

@gcusello 

>this means that you have three events but yu haven't the field Filename.

>If you run only the main search (first row) in Verbose Mode, have you this field in Interesting fields?

>probably not

Thank you, I understood that

my_search3.PNG

 

Now please return to my task

if I run:

index=my_index
| rex field=source "(?<Filename>\w+\.csv)$"
| stats count BY Filename
| append [ | inputlookup Lookup_table_with_filename1.csv | eval count=0 | fields Filename count ]
| stats sum(count) AS Total BY Filename

I see:

my_search5.PNG

I don't understand what this (second line) line is for. | rex field=source "(?<Filename>\w+\.csv)$"
Is it possible without RegEx?

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @appsik,

in the second raw I extract the filename from the source using a regex.

Anyway, regex is the only way to extract a parte or the source field.

If you don't know very well regexes I hint to use some time to learn this because regexes are very usegul in Splunk.

Ciao.

Giuseppe

0 Karma

appsik
Explorer

@gcusello 

I have the problem not with RegEx, but with the variable: Filename. Where is this variable initialized

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @appsik,

Filename isn't a variable but a field extracted using the regex from the source field in the second row of the search.

Ciao.

Giuseppe

0 Karma

appsik
Explorer

@gcusello 

rex field=source "(?<Filename>\w+\.csv)$"
? -> Zero or None
<Filename> -> Could you please explain what exactly <Filename> is causing here
w+ -> One or more letter, number, or _...
.csv)$ Name of file must end with .csv

0 Karma

appsik
Explorer

@gcusello ok, I'll try to understand it later.

Now I have to do my task.
When I have this output:

my_search5.PNG

Can I create an alert like this:
my_alert.PNG
Trigger condition: search Total != 1
Is there a more elegant solution?

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @appsik,

yes insert again the last row we deleted for troubleshooting (| where total=0) and use Results>0 as condition for your alert.

Ciao.

Giuseppe

0 Karma

appsik
Explorer

Hi @gcusello 

>(| where total=0)
yes, that makes sense
>Results>0
yes, that makes sense too

please stay available, now i try without lookup table. Thank you very much for the support

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @appsik ,

be sure I'm always here!

if one answer solves your need, please accept one answer for the other people of Community or tell me how I can help you.

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated 😉

0 Karma

appsik
Explorer

Hi @gcusello 

Thanks for the quick reply!
I am not allowed to create a new file. It is only 2-3 files. Later I can catch it with regex

What I think:

 

index=o sourcetyp=txt source="*\file1.txt"
OR "*\file2.txt"
OR "*\file3.txt" // if possible
| stats sum(source) AS total BY fileName - stats dc(source) as fileCount // if file2.txt is missing
| where total != 0

 

-> alert trigger, I need name of "file2.txt" to send by email

Sorry, I am absolute beginner here

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @appsik,

if you cannot creat a lookup (check this because it's strange!) and you have to check only few files, please try this:

index=your_index (source=*\file1.txt OR source=*\file2.txt OR source=*\file3.txt)
| rex field=source "(?<filename>\w+\.txt)$"
| stats count BY filename
| append [ | makeresults | eval filename=file1.txt, count=0 | fields filename count ]
| append [ | makeresults | eval filename=file2.txt, count=0 | fields filename count ]
| append [ | makeresults | eval filename=file3.txt, count=0 | fields filename count ]
| stats sum(count) AS total BY filename
| where total=0

Ciao.

Giuseppe

0 Karma
Get Updates on the Splunk Community!

2024 Splunk Career Impact Survey | Earn a $20 gift card for participating!

Hear ye, hear ye! The time has come again for Splunk's annual Career Impact Survey!  We need your help by ...

Optimize Cloud Monitoring

  TECH TALKS Optimize Cloud Monitoring Tuesday, August 13, 2024  |  11:00AM–12:00PM PST   Register to ...

What's New in Splunk Cloud Platform 9.2.2403?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.2.2403! Analysts can ...