topic Re: How to edit my search to identify peak hours of transactions with API Gateway purchase logs? in Splunk Search

How to edit my search to identify peak hours of transactions with API Gateway purchase logs?

tkwaller — Wed, 30 Mar 2016 15:44:02 GMT

Here's what I am trying to do. Using API Gateway purchase logs, identify peak and non-peak times. I want to send an alert when there are no transactions within 2 minutes during peak times and also send an alert when there are no transactions within 5 minutes during non-peak times.

So I started with trying to identify "peak" times. I thought I would use a time span. For this example, I was TRYING to use the 9AM hour span, but maybe that's not the best way to do this.

index=apigateway host=*agg* *purchase/carts* userName=* | transaction transactionId startswith=Inbound endswith=Out* | eval time=strftime(_time, "%Y-%m-%d %H:%M:%S,%3Q") | eval TimeofDay=if(like(time, "%9:%"), "Peak", "OffPeak") | where TimeofDay="Peak"

So step 1 would be:
What would be the best way to identify peak times of the day?

Re: How to edit my search to identify peak hours of transactions with API Gateway purchase logs?

jdunlea — Tue, 29 Sep 2020 09:13:45 GMT

If your data is running pretty real time and there are no major delays in the data coming in, why don't you just set up an alert to run every day at 10am to search the previous hour's worth of data and then do your transaction on that and then send out an alert if all the rest of the conditions are met? Same thing for non-peak times.

Another way is to search date_hour=9. You can get pretty granular with it too and look specifically for when "(date_hour=9) OR (date_hour=10 AND date_minute<31)" - Now you could see a span of 1 hour and a half.

Re: How to edit my search to identify peak hours of transactions with API Gateway purchase logs?

woodcock — Wed, 30 Mar 2016 15:58:32 GMT

I suspect that you are using poorly chosen terminology in your question. I believe when you write identify you actually mean is isolate, flag, or similar. Is that right? In other words, are you trying to discern an unknown peak time (pull it out of the data) or are you trying to exploit a known peak time (put it into the data)?

Re: How to edit my search to identify peak hours of transactions with API Gateway purchase logs?

tkwaller — Wed, 30 Mar 2016 16:25:14 GMT

So theres are going to be several parts to this. Yes when I say "identify" you say "flag". Trying to "flag" all times either Peak or Not Peak.

Re: How to edit my search to identify peak hours of transactions with API Gateway purchase logs?

tkwaller — Wed, 30 Mar 2016 16:25:26 GMT

Then I will use that to say something like if "Peak" and checkout count = 0 for 2 minutes send an alert OR
If "Not Peak" and checkout count = 0 for 5 minutes send an alert

Re: How to edit my search to identify peak hours of transactions with API Gateway purchase logs?

jdunlea — Wed, 30 Mar 2016 16:30:36 GMT

I think there needs some clarification as to whether or not you already know which hours are peak and non-peak, or if you are trying to dynamically identify a peak time from the data that is coming in.

I think from your question, you already know which times are peak and all you are trying to do is direct your transactional search and alert to look at data that pertains to that peak hour only. But I could be wrong.

Re: How to edit my search to identify peak hours of transactions with API Gateway purchase logs?

tkwaller — Wed, 30 Mar 2016 16:53:30 GMT

You are also correct, I could go about this 2 ways. For this test I chose I chose the timeframe 9AM hour, but this could be extended to be 9AM-9PM as Peak may be more than a single hour.

I COULD also dynamically try to identify a peak time from the data that is coming in. For this test I chose the first option

Re: How to edit my search to identify peak hours of transactions with API Gateway purchase logs?

woodcock — Wed, 30 Mar 2016 17:01:02 GMT

Try this:

index=apigateway host=agg purchase/carts userName=* | transaction transactionId startswith=Inbound endswith=Out | eval HourOfDay=strftime(_time, "%H%M") | eval PeakOrOffPeak=if(((HourOfDay>=900) AND (HourOfDay<=1000)), "Peak", "OffPeak") | search PeakOrOffPeak="Peak"

You can then change the if to a case with more qualifications for Peak.

Re: How to edit my search to identify peak hours of transactions with API Gateway purchase logs?

tkwaller — Wed, 30 Mar 2016 20:33:15 GMT

BOOYAH
Thank you sir, of course it was my timestamp. Just broke it down too far. Form there I can get it right
Thanks for the assistance!

Re: How to edit my search to identify peak hours of transactions with API Gateway purchase logs?

tkwaller — Mon, 04 Apr 2016 16:22:32 GMT

So I took the above search and did this:

index=java host=*byx* api_domain=purchase api_method=checkoutShoppingCart status=SUCCESS subscriber=xxx@gmail.com "app_name=Gen31Application" earliest=-5min@min latest=-2min@min| eval HourOfDay=strftime(_time, "%H%M") | eval PeakOrOffPeak=if(((HourOfDay>=900) AND (HourOfDay<=1700)), "Peak", "OffPeak") | bucket _time span=1min | stats count by _time, PeakOrOffPeak | eval Check1=if(PeakOrOffPeak = "OffPeak" AND count < 1, "Trigger", "Dont Trigger")| appendcols [search index=java host=*byx* api_domain=purchase api_method=checkoutShoppingCart status=SUCCESS subscriber=xxx@gmail.com "app_name=Gen31Application" earliest=-2min@min latest=now| eval HourOfDay=strftime(_time, "%H%M") | eval PeakOrOffPeak=if(((HourOfDay>=900) AND (HourOfDay<=1700)), "Peak", "OffPeak") | bucket _time span=1min | stats count by _time, PeakOrOffPeak | eval Check2=if(PeakOrOffPeak = "OffPeak" AND count < 1, "Trigger", "Dont Trigger")]

What this does is eval 5 mintues ago to 2 minutes ago and then eval for Peak/OffPeak and a trigger condition
THEN
the search is run again to eval 2 minutes ago to now and eval for Peak/OffPeak and a trigger condition.

What I am trying to do is run this search and eval each minute and compare to the next minute. If anytime in the 5 minute span, 2 minutes = Trigger, then send an alert.

But in testing I found that this wont work as if minute 5 and minute 4 both = Trigger, since they are both under Check1, then it will not trigger.
Maybe this is not the best way to do this...any thoughts?

Thanks!