topic Re: Show error details when errors 10% higher than previous 30 mins? in Splunk Search

Show error details when errors 10% higher than previous 30 mins?

MikeyD100 — Wed, 23 Nov 2022 04:10:19 GMT

Hi,

I want to display the error details in the last 30 mins, so they can be investigated, when the amount of errors has increased by 10% from the previous 30mins.

Search 1
This is the search for the data I want to show in the results

index=myindex source=mysource sourcetype=mysourcetype FailureReason IN ("*Error1*", "*Error2*", "*Error3*") | table ReqReceivedTimestamp, APIName, ReqUrl, ShopName, ResponseCode, FailureReason, FailureServiceCalloutResponse

Search 2
This is the search I have to work out if there are over 10% compared to the last 30 mins

index=myindex source=mysource sourcetype=mysourcetype FailureReason IN ("*Error1*", "*Error2*", "*Error3*") | timechart span=30m count as server | streamstats window=1 current=f values(server) as last30 | eval difference=server-last30 | eval percentage_change=round((difference/last30)*100,2) | eval AboveThreshold=if(round(((server-last30)/last30),4)>.10, "True", null()) | where AboveThreshold = "True" | table percentage_change

I want to understand what is the best way to get these 2 searches combined and show the table from Search 1 when Search 2 >10%

Re: Show error details when errors 10% higher than previous 30 mins?

VatsalJagani — Wed, 23 Nov 2022 06:12:11 GMT

@MikeyD100 - Sorry I haven't looked at your second search closely. I'm expecting that is working as you need and give a solution to combine these searches.

index=myindex source=mysource sourcetype=mysourcetype FailureReason IN ("*Error1*", "*Error2*", "*Error3*") | search [index=myindex source=mysource sourcetype=mysourcetype FailureReason IN ("*Error1*", "*Error2*", "*Error3*") | timechart span=30m count as server | streamstats window=1 current=f values(server) as last30 | eval difference=server-last30 | eval percentage_change=round((difference/last30)*100,2) | eval AboveThreshold=if(round(((server-last30)/last30),4)>.10, "True", null()) | eval sourcetype=if(AboveThreshold="True", "SEARCH_SOMETHING_RANDOM_RANDOM_RANDOM", "*") | table sourcetype] | table ReqReceivedTimestamp, APIName, ReqUrl, ShopName, ResponseCode, FailureReason, FailureServiceCalloutResponse

I hope this works as you expects.

Re: Show error details when errors 10% higher than previous 30 mins?

yuanliu — Wed, 23 Nov 2022 08:50:03 GMT

The second search is much too complicated for the task at hand. Combine the two with an extremely literal interpretation of your requirement.

index=myindex source=mysource sourcetype=mysourcetype FailureReason IN ("*Error1*", "*Error2*", "*Error3*") earliest=-60m | eventstats count sum(eval(if(_time < relative_time(now(), "-30m"), 1, 0))) as previous_30m | eval show_detail = if(count > previous_30m * 2.1, "yes", null()) ``` current > 1.1 * previous ==> (current + previous) > 2.1 * previous ``` | where isnotnull(show_detail) AND _time > relative_time(now(), "-30m") ``` only show current period ``` | table ReqReceivedTimestamp, APIName, ReqUrl, ShopName, ResponseCode, FailureReason, FailureServiceCalloutResponse

Re: Show error details when errors 10% higher than previous 30 mins?

MikeyD100 — Wed, 23 Nov 2022 14:35:56 GMT

Hi @yuanliu thanks, this example is much more straight forward. I have used that as guide and changed my code to meet my needs and this seems to work well now

index=myindex source=mysource sourcetype=mysourcetype FailureReason IN ("*Error1*", "*Error2*", "*Error3*") earliest=-60m | eventstats count sum(eval(if(_time > relative_time(now(), "-30m"), 1, 0))) as current_30m ```Get current``` | eventstats count sum(eval(if(_time < relative_time(now(), "-30m"), 1, 0))) as previous_30m ```Get previous``` | eval show_detail = if(round(((current_30m-previous_30m)/previous_30m),4)>0.1, "True", null()) ```Get percentage change compared to previous 30 mins``` | where isnotnull(show_detail) AND _time > relative_time(now(), "-30m") | table ReqReceivedTimestamp, APIName, ReqUrl, ShopName, ResponseCode, FailureReason, FailureServiceCalloutResponse

Re: Show error details when errors 10% higher than previous 30 mins?

VatsalJagani — Wed, 23 Nov 2022 15:15:24 GMT

@MikeyD100 - Kindly accept the answer that helped you resolve your query by clicking on "Accept as Solution", this helps other users in the community.