Splunk Search
Highlighted

How to trigger an alert when I get an event with a certain string of text, and only display newest events using the date in the log file?

Engager

Hi,

I'm trying to get alerts on Splunk every time I get a new entry with text 'No space left on device' in a log file.

Here's a sample:

April 06, 2016 10:28 -> ERROR -> Setting projects/apq8084-la-1-2-mdm9635m-tn-1-0r121032d-1cadence1synccode-system to erred, Sync error Warning: Permanently added 'host' (ECDSA) to the list of known hosts.^M
stty: standard input: Invalid argument
remote: fatal: fsync error on './objects/pack/tmppackMXkovr': No space left on device^[[K
error: unpack failed: index-pack abnormal exit

My search:

host=servername source=/usr/sync.log No space left on device latest=-24h

How do I do the search to display only the newest based on the text displayed (in the case above that will be April 06, 2016). By setting the search like above, I'm getting alerts regarding that same alert.

Thanks

0 Karma
Highlighted

Re: How to trigger an alert when I get an event with a certain string of text, and only display newest events using the date in the log file?

Communicator

Unless I'm misunderstanding your question, if you just want the latest event you can use the head command:

host=servername source=/usr/sync.log TERM(No space left on device) earliest=-24h | head 1
Highlighted

Re: How to trigger an alert when I get an event with a certain string of text, and only display newest events using the date in the log file?

Engager

Yes, I want the latest event, but how does Splunk interpret the latest?
Does the head command first finds the first occurrence of the TERM string? In this case, the log file amends the new entries at the bottom of all the contents.

Running
host=servername source=/usr/sync.log No space left on device earliest=-24h
still displays the first entries (in this example the April 06, 2016) but I haven't set it up as an Alert, I will go do that and post the result with both suggestions.

0 Karma
Highlighted

Re: How to trigger an alert when I get an event with a certain string of text, and only display newest events using the date in the log file?

Communicator

Splunk interprets the latest by giving a timestamp to every event that is ingested. By default when you search, Splunk returns the events in descending order (most recent events on top). If you have your search set for earliest=-24h but it is still returning events from April, it sounds like Splunk isn't understanding your timestamp correctly when it is being ingested. When looking at your events, does the _time field/column match the date you see in the event?

0 Karma
Highlighted

Re: How to trigger an alert when I get an event with a certain string of text, and only display newest events using the date in the log file?

Engager

No, it doesn't show the same date, if I do the search today it will show up in the left side, but the date from the log file is still April 06.

Search string:
host= source=sync.log No space left on device earliest=-24h | head 1

alt text

0 Karma
Highlighted

Re: How to trigger an alert when I get an event with a certain string of text, and only display newest events using the date in the log file?

Legend

Schedule this alert/search to run every 24 hour. It will look in the past 24 hours for the occurrence of that text. Trigger alert when count>0. You can adjust the frequency, if you want it sooner or later, just change the -24h to 1h (for example) and run the alert every hour.

host=servername source=/usr/sync.log No space left on device earliest=-24h 
0 Karma