I will like to search for a missing word like "main" on an indexed log and alert if that word is not found in the last 15 minutes
I do not want to get any other results like "everything else but main"
We generate thread dumps and if the "main" thread is missing from the dump it will indicate that the process is up but not processing.
@iggydolby2 - try something like below, based on thise search you can trigger alert if result.count >=1
index=<your index name> source=<your sourcename> |eval x= if(match(_raw,"main"), 1,0)| eventstats sum(x) as sum| where sum=0
When you define an alert, you have the option of setting the trigger condition to fire when number of results is equal to 0. Then all you need to do for your search is try to find that word in your logs, which will trigger the alert if it doesn't find it:
index=where_you_are_looking "main"
https://docs.splunk.com/Documentation/Splunk/7.2.6/Alert/AlertTriggerConditions
Thank you for the advice. Actually it is working now for me with your original search query. I'm not sure if I was doing something wrong but it works for me and already setup the alerts.
No worries, glad it is working in the end. Looks like you accepted the wrong answer as accepted though!
A simple NOT "word" will return events that do not contain that string.
index=xxxhub host="xxxxx" source=/logs/SystemOut.log NOT "Exceptions"
Alert on Number of Results is greater than 0.
@iggydolby2 - try something like below, based on thise search you can trigger alert if result.count >=1
index=<your index name> source=<your sourcename> |eval x= if(match(_raw,"main"), 1,0)| eventstats sum(x) as sum| where sum=0
This is not working.... I've tried a word that shows up on every JAVA app SystemOut.log like "Exception" and that search query shows me everything but "Exceptions" on the search results.
What I will like is a query that should only show any results if the word "Exceptions" is missing.
Here is the result of your suggested query:
index=xxxhub host="xxxxx" source=/logs/SystemOut.log |eval x= if(match(_raw,"exceptions"), 1,0)| eventstats sum(x) as sum| where sum=0
Results I only pasted a few lines but it's full of results.....
I should see NO results because the word "Exception" is NOT missing but appears multiple times....
Again, I want to see results if the "word" is missing from the log.
[5/21/19 13:03:08:696 PDT] 0000010d SAMLDefaultLo I org.springframework.security.saml.log.SAMLDefaultLogger log AuthNRequest;SUCCESS; xx.00.00.103;xxxx.com;;;
host = xxxxx.com source = /logs/SystemOut.log sourcetype = sysout
5/21/19
1:03:07.411 PM
ID: 25179
Response-Code: 200
Content-Type: application/xml;charset=UTF-8
Show all 8 lines
@iggydolby2 Please make sure you use the exact word with the proper case as match is case sensitive. This should work as per this logic it will assign value of 1 to x sum when any event contains Exception word that means sum will be >0 if there is any event which contains exact match for word Exception.
Thanks but I'm still getting results.....
it does not matter the case..... I should get NO results because "Exception" is NOT missing.
Try it yourself please and you will see..... find a word that is on every log during the time span analyzed.
Then do the query to see if you get a result if that word is missing........
@iggydolby2 I tried and I do not get any results when searching on Error , if i change my condition sum > 0 only then I am getting events. See below example it returns no events as there are events with word Error and yes it expects to be an exact match.
index="_internal" | eval x= if(match(_raw,"Error"), 1,0) | eventstats sum(x) as sum | where sum =0
This won't work because sum(x) is only going to be 0 if the string, "Error" in this case, is not present in any event. If x=1 for any event then sum is going to be greater than 0.