topic Re: Alert on a new source file? in Alerting

Alert on a new source file?

mitag — Wed, 29 Jan 2020 02:40:42 GMT

Best way to alert on a new source file for a specific sourcetype?

Bonus point if we can include the parent directory of where the source file is located.

An appliance called "AWS Elemental Live" that encodes and streams video, creates a new directory and a set of logs for every scheduled event. Something like this:

# ls -l /opt/elemental_se/web/log/10000/job_2035/
total 120
-rw-rw-r-- 1 elemental apache 36168 Jan 28 00:55 20200128T005500_emecmd.xml
-rw-rw-r-- 1 elemental apache 44368 Jan 28 02:05 20200128T005501_eme.log
-rw-rw-r-- 1 elemental apache 32955 Jan 28 02:05 20200128T005501_eme_ve.log
-rw-rw-r-- 1 elemental apache  2618 Jan 28 02:04 20200128T005501_ingest_2046.log

How do I create an alert whenever that happens?

The alert ideally should be something like this:

Splunk is reporting that "job_2035" has started recording on $host$

... (then I could try to also alert on the end of the recording if I could figure out how)

(I haven't found unique or common enough events in those logs - "recording started" or some such - so I could alert on those events rather than on the source file creation. So the alert condition must be the fact of a new source file creation.)

Thanks!

Re: Alert on a new source file?

bandit — Wed, 29 Jan 2020 04:28:15 GMT

This will look for events where first event time is under 10 minutes. You can adjust the threshold age_seconds below up/down to your liking. Adjust the index and sourcetype constraints to match your actual index and sourcetype.

| tstats min(_time) as first_time where index="*" sourcetype="*" source="*job_*" by sourcetype source 
| fields first_time index sourcetype source 
| eval current_time=now() 
| eval age_seconds=current_time-first_time 
| search age_seconds<=600 
| eval age=tostring(age_seconds,"duration") 
| eval age = replace(age,"(\d+)\:(\d+)\:(\d+)","\1h \2min \3s") 
| convert ctime(first_time) ctime(current_time) 
| table current_time first_time age age_seconds sourcetype source

Re: Alert on a new source file?

bandit — Wed, 29 Jan 2020 04:38:10 GMT

you can optionally add a field extraction for your Job id like so

| rex field=source "/(?<job_id>job_\d+)/"

Re: Alert on a new source file?

bandit — Wed, 29 Jan 2020 04:52:20 GMT

You might also tinker with the transaction command using startswith and endswith

https://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Transaction