How about this, which finds all services that have timespans with a count greater than the average plus the standard deviation for that service OR a count less than the average minus the standard deviation for that service. index=source STATUS=499 SERVICE!="Health#*" | bin _time span=1m | stats count BY SERVICE _time | eventstats avg(count) as avg_count stdev(count) AS std_dev_count max(count) AS max_count min(count) AS min_count BY SERVICE | where max_count>avg_count+std_dev_count OR min_count<avg_count-std_dev_count | timechart useother=f values(count) AS count by SERVICE ... View more

We have a lookup file that contains host, sourcetype, state, max_delay, and admin. All hosts start with state="up" . The max_delay field allows us to have hosts with varying expected delays between events. (Some hosts are expected to log events every 10 seconds, and others may go 3 hours between events in normal operations.) The admin field defines who gets notified when a given host goes down - again, this is for modularity. Every 10 minutes, we run the following search: | tstats latest(_time) as latest where index=* OR index=_* by host, sourcetype | search [ |inputlookup critical_hosts | fields host sourcetype ] | lookup critical_hosts host sourcetype OUTPUT max_delay, severity, admin, state, _key | eval current_delay=now()-latest | where max_delay<current_delay | search state="up" | eval state="down" | outputlookup hosts_lookup append=true key_field=_key | fillnull value="" admin | eval admin="[email protected],".admin | convert ctime(latest) | map search="| sendemail from=\"[email protected]\" to=\"$admin$\" subject=\"Splunk Alert: Critical Host Not Logging: $host$\" message=\"Splunk last heard from critical host $host$ with sourcetype $sourcetype$ and severity $severity$ at $latest$. The maximum expected delay from this host is $max_delay$ seconds, while the current delay is $current_delay$ seconds. Please check the host for possible failure. If you believe you received this message in error, please email [email protected]. Thank you.\" server=\"ip.address\"" maxsearches=1000 We have a similar search that runs every 10 minutes looking for hosts that have resumed reporting. By updating the lookup file with changes in state, we can send alerts only when a host appears to go down and then when it appears to come back up - not every 5 minutes while it remains down. ... View more

The accum command calculates a running total of the values in a specified field. Here's a very short run-anywhere example: | makeresults | eval a=5 | append [| makeresults | eval a=1 ] | append [| makeresults | eval a=7 ] | accum a AS b This will create a table with the _time that the events were generated, the field a (which is assigned the values 5, 1, and 7) and the field b (which contains a running total of the values from field a : 5, 6, 13). If you don't specify the AS clause, the accum command will store the running total in the source field: | makeresults | eval a=5 | append [| makeresults | eval a=1 ] | append [| makeresults | eval a=7 ] | accum a This will just give you a table with the _time field and the field a , which no longer contains the original source values but only the running total of the source values. ... View more

Assuming the date columns will always take that format, this should work: your base search | eval total_date_columns=0 | foreach 2018* [ total_date_columns=total_date_columns+1 ] ... View more

Good news - you don't need any regex for this! your base search | eval sig_names=split(sig_names, ",") | stats count by sig_names The split function will break the sig_names field into multiple values, as desired, and leave the field alone if there are no commas. ... View more

How about this: your base search | eval Physical_Linux_CPU=if(PhyicalVirtual="Physical" AND OS="linux", CPUCount, 0) | eval Physical_Windows_CPU=if(PhyicalVirtual="Physical" AND OS="windows", CPUCount, 0) | eval Virtual_Linux_CPU=if(PhyicalVirtual="Virtual" AND OS="linux", CPUCount, 0) | eval Virtual_Windows_CPU=if(PhyicalVirtual="Virtual" AND OS="windows", CPUCount, 0) | eval Physical_Linux_Memory=if(PhyicalVirtual="Physical" AND OS="linux", MemoryCount, 0) | eval Physical_Windows_Memory=if(PhyicalVirtual="Physical" AND OS="windows", MemoryCount, 0) | eval Virtual_Linux_Memory=if(PhyicalVirtual="Virtual" AND OS="linux", MemoryCount, 0) | eval Virtual_Windows_Memory=if(PhyicalVirtual="Virtual" AND OS="windows", MemoryCount, 0) | stats sum(*_CPU) AS *_CPU sum(*_Memory) AS *_Memory There's probably a cleaner way to do that using foreach , but I'm not confident about the syntax and don't have access to my Splunk instance at the moment. 🙂 But this should definitely work. ... View more

What you're describing sounds like the expected behavior to me, so maybe you can clarify. Let's say you set your timepicker to search for events "Yesterday". Then when you run this: your base search | eval _time=strptime(date,"%m/%d/%Y") | sort - _time | addinfo | where _time>=info_min_time AND (_time<=info_max_time OR info_max_time="+Infinity") | stats count as "total status" by "status" You should get back two columns: total status and status , and the number will reflect the number of events that took place yesterday. When you run this: your base search | eval _time=strptime(date,"%m/%d/%Y") | sort - _time | addinfo | stats count as "total status" by "status" you aren't filtering down to events that match your timepicker, so it makes complete sense that you would often get fewer events returned from the first search than from the second. Let's take it step by step: | eval _time=strptime(date,"%m/%d/%Y") | sort - _time | addinfo The eval statement converts a field called date into a field called _time by using the strptime function. The sort statement is sorting your events into reverse chronological order. The addinfo statement is arguably the most complex of the three lines. As per documentation, this is adding four fields: info_min_time (the earliest time defined by your timepicker), info_max_time (the latest time defined by your timepicker), info_sid (the search ID number of this search), and info_search_time (the epoch time at which the search was run). After these, in the first search, you do some filtering: | where _time>=info_min_time AND (_time<=info_max_time OR info_max_time="+Infinity") This discards all events with a timestamp before or after the window defined by your timepicker. So if your timepicker says "Yesterday", then this will discard events from before or after yesterday. Finally, you call: | stats count as "total status" by "status" This is just a counting function, but of course it will yield different counts based on whether or not you've discarded events based on when they took place. If you use the | where... statement, then you will retain fewer events for this count. ... View more