Splunk Alert when the time taken is greater than 50000 ms

[praneethlekkala]

Hi 

We would like to create a splunk alert for long running requests.

If the request exceeds 5000ms then we should get an alert.

Search Query : sourcetype="access:log" host=hostname* USERID "search"

The out put that we get is:

8/20/21
12:07:07.000 AM
30.X.X.X 7x4697381x3 USERID[20/Aug/2021:00:07:07 -0400] "POST /rest/api/2/search HTTP/1.1" 200 63 243 "-" "APP.Api/10.98.3.17993 APP/10.98.3.17993 (Microsoft Windows NT 6.2.9200.0)" "1duei1d";52.14.54.126, 30.x.x.x, 30.x.x.x https-jsse-nio-8443-exec-142
host = hostnamesource = /access_log.2021-08-20sourcetype = access

Is there a way we can accomplish this?

[praneethlekkala]

I figured out the following, can this be tweaked?

sourcetype="access_log" host=hstname* USERID "search"
| eval headers=split(_raw," ")
| eval method=mvindex(headers,5)
| eval Request=mvindex(headers,6)
| eval Status=mvindex(headers,8)
| eval Payload=mvindex(headers,9)
| eval req_time=mvindex(headers,10)
| eval uri=mvindex(headers,11)
| eval Method=replace(method,"\"","")
| eval uri=replace(uri,"\"","")
| eval RequestTime_Seconds = req_time*0.001
| eval Response_Time_in_Seconds= round(RequestTime_Seconds,2) | table Response_Time_in_Seconds host _time Request Status | where Response_Time_in_Seconds > 5

The Output i get:

Response_Time_in_Seconds host _time Request Status
1 5.89 HOSTNAME 2021-08-20 02:40:31 /rest/api/2/search 200
2 7.34 HOSTNAME 2021-08-20 04:42:25 /rest/api/2/search 200

[praneethlekkala]

@venkatasri 

30.X.X.X 7x4697381x3 USERID[20/Aug/2021:00:07:07 -0400] "POST /rest/api/2/search HTTP/1.1" 200 63 243 "-" "APP.Api/10.98.3.17993 APP/10.98.3.17993 (Microsoft Windows NT 6.2.9200.0)" "1duei1d";52.14.54.126, 30.x.x.x, 30.x.x.x https-jsse-nio-8443-exec-142
host = hostnamesource = /access_log.2021-08-20sourcetype = access

from the above its 243 is the time taken. 200 is the status and 63 is bytes sent

[praneethlekkala]

@venkatasri its in milli seconds

[venkatasri]

@praneethlekkala  can you highlight in logs which one is the response time value?

[venkatasri]

@praneethlekkala 

Can you locate what is the response time in your log and was it in seconds or milli seconds?