How can I send a notification when file is missing?

[appsik]

Hello dear community,

I am new here and hope for warm support.

The following problem I have to solve: I have several files and if a document is missing, should be sent a notification with the reference to this file.

Example:
File12324.txt
File21111.txt
Filefdfdf.txt
(naming without pattern)
If next day File21111.txt is missing, email goes out with content "..." + File21111.txt + "..."

Thanks for the advice

[appsik]

@gcusello 

if I run:

index=my_index sourcetype="csv" source=FileA.csv OR source=FileB.csv OR source=FileC.csv OR source=FileD.csv
	| rex field=source "(?<Filename>\w+\.csv)$"
	| stats count BY Filename
	| append [ | makeresults | eval filename=FileA.csv, count=0 | fields Filename count ]
	| append [ | makeresults | eval filename=FileB.csv, count=0 | fields Filename count ]
	| append [ | makeresults | eval filename=fileC.csv, count=0 | fields Filename count ]
	| append [ | makeresults | eval filename=fileD.csv, count=0 | fields Filename count ]
	| stats sum(count) AS Total BY Filename

I see only A,B and C

D is not available and must be displayed with 0

[gcusello]

Hi @appsik ,

sorry: field names are case sensitivi: use "Filename" also in the four appends

index=my_index sourcetype="csv" source=FileA.csv OR source=FileB.csv OR source=FileC.csv OR source=FileD.csv
	| rex field=source "(?<Filename>\w+\.csv)$"
	| stats count BY Filename
	| append [ | makeresults | eval Filename=FileA.csv, count=0 | fields Filename count ]
	| append [ | makeresults | eval Filename=FileB.csv, count=0 | fields Filename count ]
	| append [ | makeresults | eval Filename=fileC.csv, count=0 | fields Filename count ]
	| append [ | makeresults | eval Filename=fileD.csv, count=0 | fields Filename count ]
	| stats sum(count) AS Total BY Filename

Ciao.

Giuseppe

[appsik]

Hi @gcusello 

After I have executed the script, I don't see MissingFile.scv

index=my_index source=FileA.csv OR source=FileB.csv OR source=FileC.csv OR source=MissingFile.csv
	| rex field=source "(?<Filename>\w+\.csv)$"
	| stats count BY Filename
	| append [ | makeresults | eval Filename=FileA.csv, count=0 | fields Filename count ]
	| append [ | makeresults | eval Filename=FileB.csv, count=0 | fields Filename count ]
	| append [ | makeresults | eval Filename=FileC.csv, count=0 | fields Filename count ]
	| append [ | makeresults | eval Filename=MissingFile.csv, count=0 | fields Filename count ]
```	| stats sum(count) AS Total BY Filename```

Something is missing here

[gcusello]

Hi @appsik,

please try this:

index=my_index source=FileA.csv OR source=FileB.csv OR source=FileC.csv OR source=MissingFile.csv
	| rex field=source "(?<Filename>\w+\.csv)$"
	| stats count BY Filename
	| append [ | makeresults | eval Filename="FileA.csv", count=0 | fields Filename count ]
	| append [ | makeresults | eval Filename="FileB.csv", count=0 | fields Filename count ]
	| append [ | makeresults | eval Filename="FileC.csv", count=0 | fields Filename count ]
	| append [ | makeresults | eval Filename="MissingFile.csv", count=0 | fields Filename count ]
	| stats sum(count) AS Total BY Filename

Ciao.

Giuseppe

[appsik]

@gcusello Thank you very much, this is a good solution. Very helpful as a beginner

[gcusello]

Hi @appsik ,

if one answer solves your need, please accept one answer for the other people of Community or tell us how we can help you.

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated by all the Contributors;-)

[appsik]

Hi @gcusello after run

index=my_index sourcetype="csv" source=FileA.csv OR source=FileB.csv OR source=FileC.csv OR source=FileD.csv
	| rex field=source "(?<Filename>\w+\.csv)$"
	| stats count BY Filename
	| append [ | makeresults | eval Filename=FileA.csv, count=0 | fields Filename count ]
	| append [ | makeresults | eval Filename=FileB.csv, count=0 | fields Filename count ]
	| append [ | makeresults | eval Filename=fileC.csv, count=0 | fields Filename count ]
	| append [ | makeresults | eval Filename=fileD.csv, count=0 | fields Filename count ]
	| stats sum(count) AS Total BY Filename

I still see only A,B,C

[PickleRick]

Your eval commands are wrong. If you say

| eval key=value

You're telling Splunk to assign a value of a field named "value" to the field named "key".

So if you do

| eval value=4
| eval key=value

The "key" field will have value of 4.

But if you need to assign a literal value of "value" (in your case "FieldA.csv"), you need to enclose that value in quotes.

| eval Filename="FieldA.csv"

Otherwise, if you omit the quotes, Splunk will try to find an existing field called "FieldA.csv" and assign its value to the field "Filename". Since you don't have such field defined, the resulting field "Filename" will be null.

[appsik]

@gcusello 

I would like to better understand the background processes and have installed Splunk on my Windows machine.
I am now wondering how to get from my home: C:\Program Files\Splunk
Access any file via search console:

I created index:

I created the following files:
C:\Program Files\Splunk\worldcities.csv
C:\Program Files\Splunk\worldcities1.csv
C:\Program Files\Splunk\worldcities2.csv

In the search console

index=my_index sourcetype="csv" source="worldcities.csv"
| stats dc(source) as fileCount

And I have fileCount: 0, so the file does not exist

[gcusello]

Hi @appsik,

did you configured the input ro read and index these files?

It isn't suffient to create the index and run the search, you have to:

• ingest the sources,
• parse them, choosing a correct sourcetype (in your case csv or a custom one),
• and then index them,
• at this point you can search them in the created index

For more infos see https://docs.splunk.com/Documentation/SplunkCloud/8.0.1/Data/Getstartedwithgettingdatain

there are also many videos in the Splunk YouTube Channel that describe this process: https://www.youtube.com/@Splunkofficial 

Ciao.

Giuseppe

[PickleRick]

It doesn't work that way. Splunk doesn't simply process any files you throw into its directory.

See the introduction here: https://docs.splunk.com/Documentation/Splunk/9.0.4/Data/WhatSplunkcanmonitor

[appsik]

Hallo @PickleRick 

I have already uploaded my files: file1.csv, file2.csv, file3.csv as described here: https://community.splunk.com/t5/Getting-Data-In/How-to-index-a-CSV-file-from-a-local-machine/m-p/232...

[PickleRick]

OK. If you uploaded the file via GUI, as described in that article, it should be _somewhere_. Question is what sourcetype did you give it and whether it got properly timestamped and such.

Since it's - as I understand - your small testing installation, verify where your events are.

| tstats min(_time) as earliest max(_time) as latest count where index=* by index source sourcetype
| convert ctime(earliest) as earliest ctime(latest) as latest

Run this search over "all time" time range.

[appsik]

@PickleRick Thank you, now I understand how the assignment works

We have a folder in production where it automatically looks, so I thought Splunk automatically looked in the home folder. That was a misunderstanding.