I have identified an issue with a response time stats report that was built by a former Splunk specialist at my organization and I'm having trouble identifying the root cause or developing a better solution. The goal is to produce a stats table where the first column defines the range (in seconds) and the second column displays a count of transactions that occurred in that range. However, it seems that the calculations do not align with my own check of the raw data which I made in Excel — I feel the ranges must be incorrectly defined. The ranges to include in the table are as follows: <1.0 sec (next column will include a count of transactions which have a response time value of between 0 - 1.0) <2.0 sec (next column will include a count of transactions which have a response time value of between 0 - 2.0) <3.0 sec (next column will include a count of transactions which have a response time value of between 0 - 3.0.. etc) <4.0 sec <5.0 sec <6.0 sec <7.0 sec <8.0 sec <9.0 sec <=10.0 sec The search query is as follows — it is line 2 that I can't get my head around and feel it may be incorrect — it seems to be rounding values up and this is not appropriate, as we are dealing with hard range cutoffs, I.E, 1.0 seconds, 2.0 seconds, etc: eventstats count as "total" | eval in_range=round(case(responseTime<10, ceil(responseTime), responseTime>=10.0,10.0),1) | streamstats count as cnt avg(responseTime) as run_avg | stats first(total) as total last(run_avg) as run_avg max(cnt) as count count as cnt by in_range | sort 0 in_range | eval range=if(in_range>=10, ">= 10.0 sec","< "+tostring(in_range)+" sec") | eval run_avg=round(run_avg,1) | rename cnt as "No of Transactions"| table range "No of Transactions" The result of this search is a table which appears to have the correct format, however the "No of transactions" values do not seem to correctly fall within the ranges defined. Second part to the problem - optional: In addition to this, the ranges are not cumulative - ie, the actual ranges which it seems to be reporting are 0-1 sec, 1-2 sec, 2-3 sec, etc ... View more

I am attempting to set up an Alert which will trigger when average response times for various products over the week have increased by at least double in comparison to the previous week. However it is not working out exactly as I had in mind. My search query for the alert is as follows; source="transactionLog" type="report" earliest=-1d@d latest=now | eval Day=if(_time>=relative_time(now(),"@d"),"Today","Yesterday") | chart avg(responsetime) over product by Day And then I am using a custom trigger condition as follows; search where Today>2*Yesterday However the problem is, whether I add the where clause to the end of my search or not, there are still over 700,000 events returned as results - so my alert notification returns all response times for ALL products (even the ones which did not see an increase). ie; whether I include the where clause at the end of my search or not, there is still the same number of returned events? This means the alert notification contains a whole lot of irrelevant data - I would ideally like to see ONLY the instances in the alert notification where the average response time has doubled, not all of the data. I assume the WHERE clause does not actually filter out results which do not match the clause? Is there a more suitable way to approach this? ... View more

I am working with a set of transactions data where in each transaction could relate to any of our numerous systems/products. The data could be simplified for the sake of this question to contain fields similar to the below: Event 1: TransationID: 1000 Product: System1 ResponseTime: 1234 Event 2: TransactionID: 1001 Product: System2 ResponseTime: 4321 Event 3: TransactionID: 1002 Product: System3 ResponseTime: 5678 Etc. So I am interested in setting up an alert which works in the following manner: The search query will return the average response time for each product for yesterday, and compare that to the average response time for each product today and for any products where there has been an X% increase - send an email alert - If possible, the alert will advise the product in question which has experienced the fluctuation in response time. I feel like perhaps a custom trigger condition may be suitable for this instance? The search query below will successfully provide me with the average response times per product for yesterday: source="transactionLog" type="report" earliest=-1d@d latest=@d | stats avg(responsetime) by product I saw a slightly similar question on Splunk Answers in which the person was attempting to use a subsearch in order to generate his alert query - however my situation differs as I am breaking down by products, so I am not sure that I can use a Number of Results trigger condition. UPDATE: I have put together the below query - I feel this is suitable as it actually compares the responsetime - I am having trouble understanding the logic from some of the comments in the answers below. source="transactionLog" type="report" earliest=-1w@w latest=@w | stats avg(responsetime) as Previous_Week_Response | appendcols [search source="transactionLog" type="report" earliest=@w latest=now | stats avg(responsetime) as Current_Week_Response] | eval Week=if(_time>=relative_time(now(),"@w"),"Current_Week","Previous_Week") | chart avg(responsetime) over product by Week | where Current_Week_Response>=2*Previous_Week_Response ... View more