I have an alert that can clear in the same minute that it originally fired.  When the correlation search runs, both events are in it, the alert and the clearing alert.  The correlation search creates notable events for each but uses the current time for the _time for the notable events and not the _time from the original alerts.  Since both alerts are converted into notable events during the same correlation search run, they get the exact same timestamp.  This causes ITSI to not definitely know the correct order of the events and it sometimes thinks the Normal/Clear event came BEFORE the original alert. This seems odd to me.  I would have imagined that ITSI would use the original event time as the _time for the notable event but it doesn't. Any ideas on how to address?    ... View more

I'm trying to understand how to update the severity of a notable event when a new event arrives with a normal severity.  I'm feeding external alerts into ITSI and a correlation search turns it into a notable event.  I'm using a specific ID for the "Notable Event Identifier Fields".  These alerts correctly turn into notable events and placed into an episode.  When the same alert comes into ITSI, but with a "Normal"\2 severity, I expect it to change the severity of the prior notable event in the episode.  Instead, it will treat it like a new notable event and put it into the same episode.  I thought ITSI uses the Notable Event Identifier Fields to determine if two events are the same or not.  I checked that both the original event and the "clearing" event have the exact same event_identifier_hash, so why does ITSI treat it like an additional alert\event in the episode?  Instead of having one normal\clear event in the episode, I now have one critical and one normal. How are you supposed to update the status of an alert\notable event in an episode when a clearing event is received? ... View more

This sounds more complicated than it really is. At the root, I want to know if web traffic is balanced across my web farm. This query accomplishes that by finding out the average traffic per server in the farm (my sub search) and then calculate how much each server varies from that average. It works pretty well. host=myservers index=iis | stats count AS server_value BY host | join [search host=myservers index=iis | stats count as LatestValue by host | stats mean(LatestValue) as client_mean] | eval percent_variance=round(((abs(server_value-client_mean)/client_mean)*100), 0) My challenge is that I want to use this metric in Splunk ITSI. In ITSI your metrics have to have a _time associated with your events. This is easy enough to do by creating bins, doing stats by time, and joining on the _time. host=myservers index=iis | eval host=upper(host) | bin span=5m _time | stats count AS server_value BY _time, host | join _time [search host=myservers index=iis | eval host=upper(host) | bin span=5m _time | stats count as LatestValue by _time, host | stats mean(LatestValue) as client_mean by _time] | eval percent_variance=round(((abs(server_value-client_mean)/client_mean)*100), 0) This technically works but I'm running up against a classic "bin" issue. Bin always spans the bin to the whole time segments. ITSI runs metrics at 5-minute intervals by default and that makes sense. If I don't put a span, it will default to 5-second spans. That is too small of a sample time and the variances wildly differ. The 5-minute span above works well if you have a perfect 5-minute interval but you typically get some partial 5-minute block. Whatever time span I use I still typically have a partial bin that messes up the accuracy of my metric. Any ideas for solutions or alternative approaches? ... View more

I have the same problem that is unanswered here I'm trying to do stats on the last 10 minutes of data by two separate 5 minute buckets. My command looks like this: host=servers* index=iis | bin _time bins=2 | stats count as Request by _time, host When I run this for the last 10 minutes I expect 2 separate 5 minute buckets. If my query starts on an even 5 minute interval like 1:05-1:15, it works fine but the buckets expect even 5 minute increments and if I run 1:02-1:12, it will create a "1:00" bucket for the requests between 1:00 and 1:05, a "1:05" bucket for the 1:05-1:10 requests, and a "1:10" bucket for the 1:10-1:15 requests. That is 3 buckets of different sizes. Using the span=5 minutes doesn't help either. I tried just using the epoch number like: eval Time=_time | bin Time bins=2 I was trying to stop Splunk from treating the _time field specially but it put everything into a single bin. How can I get Splunk to just create 2 evenly sized/spanned buckets? ... View more

This is driving me nuts because I use strptime all the time and have many of my own working examples to reference. I was having a problem doing strptime with a more complex date that wasn't working so I kept making it more simple until even this isn't working. ... | eval TestYear="2017" | eval TestResult=strptime(TestYear,"%Y") | table TestYear, TestResult Why isn't TestResult getting the epoch time for the year? The field is not being created. This is so simple that I'm clearly doing something dumb that I'm just too close to see. Thanks guys ... View more

I have transaction records that are pretty clear. OperationType=singon Client=abc IsSuccess=1 OperationType=changePassword Client=xyz IsSuccess=0 ... Based on these records I want to see the success % rate by OperationType and Client sourcetype=mydata | stats count as Total, sum(IsSuccess) as Succeeded by OperationType, Client | eval SuccessRate=(Succeeded/Total)*100 | fields - Total, Succeeded That outputs: OperationType | Client | SuccessRate signon | abc | 95 signon | xyz | 97 changepassword| abc | 89 changepassword| xyz | 93 This creates a lot of rows, one for every Client, OperationType combination. Instead, I'd like the chart to have a column for every client and a row for each OperationType like this: OperationType | abc | xyz signon | 95 | 97 changepassword| 89 | 93 Any idea how to approach this? ... View more

I would like let a user use a timepicker to look at a problem period in a timechart and then I'd like to display another timechart below it with the same data but a week earlier for comparison. I can't put the data in the same timechart because there are too many metrics to display. How can I make the bottom panel use an offset from the timepicker? I've tried to modify the XML like this: <earliest>$tokenTIme.earliest$-7d</earliest> <latest>$tokenTIme.latest$-7d</latest> I've also tried to modify the search by doing something like this: index=a source=b earliest=$tokenTIme.earliest$-7d latest=$tokenTIme.latest$-7d Any ideas? ... View more