I have an issue where logs contain timestamps in zulu and the server uses local time for its index.  I need to calculate delays now since the logs are initially written on vendor servers and there can be between 0 delay or over an hour behind. I can't find any functions that do this and am not having luck either converting a timestamp from one timezone to another or converting timezones to seconds to do any math on. timestamp log was written   2020-07-22T12:59:12.301063Z   timezone log was indexed from _time   -0400   I am not an admin and have no control or influence over the servers or its configuration. Is there an easy way to do this inline in the query? ... View more

I need to create an alert that's more intelligent and based on a baseline. I have a search that produces the following dataset in run anywhere spl: |makeresults 1 | eval service="placeOrder", week=1, Volume=100, VolumeMed=100, VolumeLowerBound=28.75, VolumeIQR=47.5, VolumeUpperBound=175.25, VolumeOutlier=0, SuccessRate=80, FailureRate=20, RespTimeMed=500 | append [|makeresults 1 | eval service="placeOrder", week=2, Volume=95, VolumeMed=100, VolumeLowerBound=28.75, VolumeIQR=47.5, VolumeUpperBound=175.25, VolumeOutlier=0, SuccessRate=10, FailureRate=90, RespTimeMed=11400] | append [|makeresults 1 | eval service="placeOrder", week=3, Volume=105, VolumeMed=100, VolumeLowerBound=28.75, VolumeIQR=47.5, VolumeUpperBound=175.25, VolumeOutlier=0, SuccessRate=85, FailureRate=15, RespTimeMed=450] | append [|makeresults 1 | eval service="placeOrder", week=4, Volume=100, VolumeMed=100, VolumeLowerBound=28.75, VolumeIQR=47.5, VolumeUpperBound=175.25, VolumeOutlier=0, SuccessRate=75, FailureRate=25, RespTimeMed=550] | append [|makeresults 1 | eval service="placeOrder", week=5, Volume=15, VolumeMed=100, VolumeLowerBound=28.75, VolumeIQR=47.5, VolumeUpperBound=175.25, VolumeOutlier=1, SuccessRate=75, FailureRate=25, RespTimeMed=450] | fields service, week, Volume, VolumeMed, VolumeLowerBound, VolumeIQR, VolumeUpperBound, VolumeOutlier, SuccessRate, FailureRate, RespTimeMed There are 5 total KPIs: Volume, SuccessRate, WarningRate, FailureRate, ResponseTime. I want to remove outliers prior to calculating the standard deviation but the problem is a single row may only contain an outlier in one of these KPIs, so the row must stay. In this case, each KPI outlier is on a different row, so i need the StDev of Volume to exclude week 5, and I need the StDev of SuccessRate to exclude week 2, but week 2 must still be included for the Volume StDev because it is not a Volume outlier, only a SuccessRate outlier. I tried adding this to the end of the search: | stats stdev(Volume), stdev(case(VolumeOutlier=0, Volume)) as VolumeStdDev by service But, this is the result: service stdev(Volume) VolumeStdDev placeOrder 38.1772 The column where I try to limit what rows it uses is blank. VolumeStdDev should be 4.0824. How can I achieve excluding certain outlier rows of one KPI from a StDev calculation but whose row must remain to be included in StDev where it is not an outlier for those other KPIs? ... View more

I have a Time selector. Each time it's clicked, a certain set of tokens must always recalculate, including one which determines the span of time in between earliest and latest. I have 2 panels. Only 1 panel must be shown at a time, depending on how long the span is between earliest and latest. Within 1 day, show the "comparison" panel, longer than 1 day, show the "single" panel The xml below always sets both tokens, "showPanelSingle" and "showPanelComparison", but I can only have 1 set at a time. <input type="time" token="time" searchWhenChanged="false"> <label>Time Frame</label> <default> <earliest>-1d@d</earliest> <latest>@d</latest> </default> <change> <eval token="time.earliest_epoch">if(isnum('earliest'),'earliest',relative_time(now(),'earliest'))</eval> <eval token="time.latest_epoch">if(isnum('latest'),'latest',relative_time(now(),'latest'))</eval> <eval token="time_difference">$time.latest_epoch$-$time.earliest_epoch$</eval> <eval token="form.span">if($time_difference$&gt;2592000,"1d",if($time_difference$&gt;86400,"1h","1m"))</eval> <eval token="showPanel">if($time_difference$&gt;86400,"Comparison","Single")</eval> <eval token="showPanelSingle">if($showPanel$="Single","true","")</eval> <eval token="showPanelComparison">if($showPanel$="Comparison","true","")</eval> </change> </input> The xml below never sets either token, even after selecting the Time. <form theme="dark"> <label></label> <init> <condition match="$showPanel$, &quot;Single&quot;"> <eval token="showPanelSingle">true</eval> </condition> <condition match="$showPanel$, &quot;Comparison&quot;"> <eval token="showPanelComparison">true</eval> </condition> </init> Note, the Time calculations must always run, so I can't add them to a condition, but I need condition on the rest, and Splunk doesn't allow this hybrid approach, nor is it allowed to qualify multiple condition tags. How can I accomplish where 1 and only 1 of them is set (and the correct one) upon clicking "Submit" in the input filters. ... View more

I need to create volume-base alerts so we know when volume drops. The services we need to monitor are usually suffixed with its version (e.g. placeOrder is actually placeOrderv1, placeOrderv2, etc) which often ramp up and down while servicing their respective percentage of the total traffic, causing our volume based alerts to fire even though the total calls for the overall feature are the same. I chopped off the suffix and created a calculated "group" field and want to use that, but this leaves a problem when the service isn't firing at all. the volume becomes 0, but so does the "group", or total, volume, for that row. here is the half way point of my larger query so you can see the structure that is built as i append 2 results: the current, and the 6 week average, in run-anywhere query. | makeresults 1 | eval group="placeOrder" | eval service="placeOrder_v1" | eval groupVolume="375" | eval volume="175" | append [| makeresults 1 | eval group="placeOrder" | eval service="placeOrder_v3" | eval groupVolume="375" | eval volume="200"] | append [| makeresults 1 | eval group="placeOrder" | eval service="placeOrder_v1" | eval pastGroupVolume="325" | eval pastVolume="200"] | append [| makeresults 1 | eval group="placeOrder" | eval service="placeOrder_v2" | eval pastGroupVolume="325" | eval pastVolume="100"] | append [| makeresults 1 | eval group="placeOrder" | eval service="placeOrder_v3" | eval pastGroupVolume="325" | eval pastVolume="25"] | fields _time, group, service, groupVolume, volume, pastVolume, pastGroupVolume this gives the following table group service GroupVolume Volume PastGroupVolume PastVolume placeOrder placeOrder_v1 375 175 placeOrder placeOrder_v3 375 200 placeOrder placeOrder_v1 325 200 placeOrder placeOrder_v2 325 100 placeOrder placeOrder_v3 325 25 When I zip them with this command added added to it | stats values(groupVolume) as groupVolume, values(volume) as volume, values(pastGroupVolume) as pastGroupVolume, values(pastVolume) as pastVolume by group, service I get this group service GroupVolume Volume PastGroupVolume PastVolume placeOrder placeOrder_v1 375 175 325 200 placeOrder placeOrder_v2 325 100 placeOrder placeOrder_v3 375 200 325 25 _v2 is no longer being called, while _v1 is ramping down, and _v3 is ramping up. An alert will fire for _v2 because the current group volume is 0 as i compare it to the historical avg. it needs to acquire 375 because it's the same group. How can i carry over this value into the row for _v2? I need it to show this group service GroupVolume Volume PastGroupVolume PastVolume placeOrder placeOrder_v1 375 175 325 200 placeOrder placeOrder_v2 375 325 100 placeOrder placeOrder_v3 375 200 325 25 ... View more