How to generate an a alert when a specific source file is not indexed in the last 24 hours?

[deepthi5]

Hi Team ,

I have 50 source files in a folder that are getting indexed.

I need to generate an alert whenever a particular source file is not indexed and did not generate an event in the last 24 hours

Thanks and Regards,
Deepthi Bulusu

[richgalloway]

Perhaps this search will get you started.

index=foo sourcetype=bar host=baz | dedup source 

Schedule this to search the previous 24 hours and have it send an alert if the event count is less than 50. It won't tell you which source is missing, but you'll know that at least one of them is absent. If you want to know which source is missing, try Sebastian2's answer.

[Sebastian2 · Accepted solution]

You could use a lookup to verify your indexed data. Do something like that:

1. Create an CSV as lookup-table

The file column is optional here and just for a better overview. Create Lookup-Table, here as an csv for instance: etc/system/lookups/my_indexed_files.csv

source,file
yoursource1,app_1.log
yoursource2,app_2.log
[...]

Create Lookup Stanza in etc/system/local/transforms.conf

[my_indexed_files]
filename = my_indexed_files.csv
min_matches = 1
max_matches = 1
default_match = UNKNOWN INDEXER

To just check if there was any events in the last 24h hours would be risky since you woudln't notice if an file (source) is missing completely. (restart splunk now if changes were made with CLI).

2. Develope search
The idea is to get all lines from your csv, left join the actual events from the last 24h and filter only those without any events. Your search should finally look something like that:

| inputlookup my_indexed_files | join type=left source [ search index=yourindex sourcetype=yousourcetype earliest="-24h" | stats count by source ] | where isnull(count)

Test your search and check if the results are as desired, save as alert and setup the alert as required. The only disadvantage of this method is, that you will have to maintain the lookup-table etc/system/lookups/my_indexed_files.csv. How ever, on the other hand you have the advantage that you will get an alert even when a lockfile is completely deleted and theirefore not indexed anymore.

[Sebastian2]

Do you have any further information?
- Are all 50 files of the same sourcetype and/or indexed into the same index?
- Do you need an alert when a source file has not been indexed in the last 24h hours (indexer broken) or when there weren't any events indexed the last 24h hours (application writing the log files may crashed, forwarder crashed)?
- How can the events be differenced? By host? By sourcetype? By ip?

[deepthi5]

Hi,

Yes all the 50 files are of same sourcetype that is csv Sourcetype= csv same index i have created a seperate index for this xxxx

Yes i need an alert any of the source file from this 50 files did not indexed in the last 24 hours

I want the events to differenced by their sources because host,sourcetype and ip are same for all these 50 files becoz i am indexing them from local machine itself

Thanks,
Deepthi Bulusu