I have an alert set up to run every hour to look for any latency of :45 minutes. If over that send a "Please Investigate" message Index=...  | stats count max(_time) as lastTime by host | eval now=now() | eval timedelta=round((now-lastTime)/60/60,2) | eval timedelta=if(timedelta > .75,"Please Investigate", timedelta) | convert ctime(lastTime) ctime(now) | sort - timedelta The problem is that I get this alert email even when the latency is 0.00. What I really need is for the alert to trigger and run when it sees the phrase "Please Investigate" . I have been unsuccessful in setting this up in the Splunk Alert GUI as a trigger.   ... View more

I need help breaking the following data into segments. The data is currently lumped together. I have been working with the Splunk Add Data feature to attempt to parse the data correctly 07400 16:31:30.320 Processing 51 log entries in <servername.615494dd0000.dblog> from servername 07784 16:31:30.492 Processing 51 log entries in <servername.615494e00000.dblog> from servername 07400 16:31:30.633 DBLog Summary: time=313ms (total=51, mean time=6.137/rec), Message:(c=32, t=297) Content:(c=5, t=0) NodeStats:(c=1, t=0) VirusScannerStats:(c=13, t=0) 07784 16:31:30.987 DBLog Summary: time=484ms (total=51, mean time=9.490/rec), Message:(c=35, t=469) Content:(c=4, t=0) NodeStats:(c=1, t=0) VirusScannerStats:(c=11, t=0) 07784 16:31:31.213 Processing 51 log entries in <servername.615494e00000.dblog> from servername 07784 16:31:31.278 DBLog Summary: time=62ms (total=51, mean time=1.216/rec), Message:(c=31, t=31) Content:(c=9, t=16) NodeStats:(c=1, t=0) VirusScannerStats:(c=10, t=0) 07784 16:31:31.691 Processing 51 log entries in <servername.615494e20000.dblog> from servername 07400 16:31:31.739 Rule Profiler: writing queued records to the database. 07400 16:31:31.745 Rule Profiler: finished writing queued records to the database. Record count: 53 07784 16:31:31.776 DBLog Summary: time=93ms (total=51, mean time=1.824/rec), Message:(c=31, t=78) Content:(c=6, t=0) NodeStats:(c=2, t=0) VirusScannerStats:(c=12, t=0) In Regex tester I have used the regex (\d{5}\s+\d{2}:\d{2}:\d{2}.\d{3}\s+Processing 51) to correctly capture where the data needs to be on a new line. I need the event data parsed to look as follows: 07400 16:31:30.320 Processing 51 log entries in <servername.615494dd0000.dblog> from servername 07784 16:31:30.492 Processing 51 log entries in <servername.615494e00000.dblog> from servername 07400 16:31:30.633 DBLog Summary: time=313ms (total=51, mean time=6.137/rec), Message:(c=32, t=297) Content:(c=5, t=0) NodeStats:(c=1, t=0) VirusScannerStats:(c=13, t=0) 07784 16:31:30.987 DBLog Summary: time=484ms (total=51, mean time=9.490/rec), Message:(c=35, t=469) Content:(c=4, t=0) NodeStats:(c=1, t=0) VirusScannerStats:(c=11, t=0) 07784 16:31:31.213 Processing 51 log entries in <servername.615494e00000.dblog> from servername 07784 16:31:31.278 DBLog Summary: time=62ms (total=51, mean time=1.216/rec), Message:(c=31, t=31) Content:(c=9, t=16) NodeStats:(c=1, t=0) VirusScannerStats:(c=10, t=0) 07784 16:31:31.691 Processing 51 log entries in <servername.615494e20000.dblog> from servername 07400 16:31:31.739 Rule Profiler: writing queued records to the database. 07400 16:31:31.745 Rule Profiler: finished writing queued records to the database. Record count: 53 07784 16:31:31.776 DBLog Summary: time=93ms (total=51, mean time=1.824/rec), Message:(c=31, t=78) Content:(c=6, t=0) NodeStats:(c=2, t=0) VirusScannerStats:(c=12, t=0) I have tried LINE_BREAKER=([\r\n]+), BREAK_ONLY_BEFORE, MUST_BREAK_AFTER, MUST_NOT_BREAK_BEFORE along with using the regex shown above in the Splunk Wizard will not break the data as needed. Thanks ... View more

I have seen a lot of questions and answers concerning fillnull and fillnull=0. However, I am unable to get my search to trigger an alert when the message appears in the GUI, "No results found". Based on the the time the log files are coming into our server, the timestamp of the events and when the the search is run I am seeing that message. I need it to trigger if I can capture that message or if the count=0 sourcetype=,mysourcetype | rex field=source "^[^.\n]\/(?P[^_]+)" | chart count by server | append [| gentimes start=-1 | eval server="" | table server | makemv server | mvexpand server ] | fillnull | stats max() as * by server ... View more

The field abc will list servers multiple times in the realtime log file. I created a inputlookup file (abc_servers.csv) and search to compare what is in the log file to the inputlookup. The inputlookup abc_sources.csv I created has 43 entries. However, when I run the following search I may see a stats count for 41 abc and does not list the other 2 missing with a value of 0 and thus not alert me of the 0 count. I also created an outputlookup (tmp_abc.csv) to give a base line but I am not sure how to compare it to the inputlookup to determine if there is a 0 count sourcetype="mysourcetype:" abc= | search [inputlookup abc_servers.csv] | stats count by abc | fillnull value=0 ... View more