How can I find the response time with an event time and max time of the events

[slearntrain]

We have a use case where we need to calculate the time difference between the maximum infotime (steptype="endNBflow") and infotime where steptype is "end payload". This particular message has 16 events comprising request and response flows. Request flow ends with "end Payload" and response flow ends with steptype "end NB Flow".

I have the below query:

index="xyz" sourcetype=openshift_logs openshift_namespace="qaenv" "a9ecdae5-45t6-abcd-35tr-6s9i4ewlp6h3"
| rex field=_raw "\"APPID\"\:\s\"(?<appid>.*?)\""
| rex field=_raw "\"stepType\"\:\s\"(?<steptype>.*?)\""
| rex field=_raw "\"flowname\"\:\s\"(?<flowname>.*?)\""
| rex field=_raw "INFO ((?<infotime>\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2},\d{3}))" | sort infotime
| table appid,flowname, steptype, infotime

How can I retrieve the value what I am looking for. Any guidance here would be much appreciated.

[gcusello]

Hi @slearntrain,

you have to use stats instead table:

index="xyz" sourcetype=openshift_logs openshift_namespace="qaenv" "a9ecdae5-45t6-abcd-35tr-6s9i4ewlp6h3"
| rex field=_raw "\"APPID\"\:\s\"(?<appid>.*?)\""
| rex field=_raw "\"stepType\"\:\s\"(?<steptype>.*?)\""
| rex field=_raw "\"flowname\"\:\s\"(?<flowname>.*?)\""
| rex field=_raw "INFO ((?<infotime>\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2},\d{3}))" 
| stats
   latest(eval(if(steptype="endNBflow"))) AS endNBflow
   latest(eval(if(steptype="end payload"))) AS endPayload
   BY appid flowname
| eval diff=endPayload-endNBflow

Ciao.

Giuseppe

[slearntrain]

Hi gcusello, 

Thank you for responding. unfortunately, I didn't quite get what I was looking for. Initially, when I ran your query, I got the error that if command does not meet the requirement. So I had to add the true/false parameters.

As I am looking for infotime for each of the steptypes, I added in the true section. -- "latest(eval(if(steptype="endNBflow", infotime,0)))"

Now, I need to find the "diff" (responseTime) in the table equivalent to the appid as that is the response time for the message.

I have modified your query based on the time formats that I want:

index="xyz" sourcetype=openshift_logs openshift_namespace="qaenv" "a9ecdae5-45t6-abcd-35tr-6s9i4ewlp6h3"
| rex field=_raw "\"APPID\"\:\s\"(?<appid>.*?)\""
| rex field=_raw "\"stepType\"\:\s\"(?<steptype>.*?)\""
| rex field=_raw "\"flowname\"\:\s\"(?<flowname>.*?)\""
| rex field=_raw "INFO ((?<infotime>\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2},\d{3}))" 
| stats
   latest(eval(if(steptype="endNBflow", infotime,0))) AS endNBflow
   latest(eval(if(steptype="end payload",max(infotime),0))) AS endPayload
   BY appid flowname|eval endNBflowtime=strptime(endNBflow,"%Y-%m-%d %H:%M:%S,%3N")| eval endPayLoadtime=strptime(endPayLoad,"%Y-%m-%d %H:%M:%S,%3N")|eval diff=endpayloadtime-endNBflowtime|eval responseTime=strftime(diff,"%Y-%m-%d %H:%M:%S,%3N")

But now how to bring the response time in the table format corresponding to the appid?

[gcusello]

Hi @slearntrain,

yes, sorry: I forgot the second part of the if statements, but now it's correct.

what's the format you would for the results?

With this earch you have in the same row:

• appid
• flowname
• endNBflow
• endPayload
• diff

if you want to have in different rows endNBflow and endPayload, where do you want to put the difference?

Could you indicate how would you have the results?

Ciao.

Giuseppe

[slearntrain]

Hi Giuseppe,

I want to view the results in the below format. I also want the diff time in human readable format like 10sec, 15 mins etc.

Appid

Responsetime(Diff)

In my usecase- I have more that 5000 messages, each successful message has 16 steptypes, so I have put the query in this way.-

index="abc" sourcetype=openshift_logs openshift_namespace="qaenv" "a9ecdae5-45t6-abcd*"
| rex field=_raw "\"Application-ID\"\:\s\"(?<appid>.*?)\""
| rex field=_raw "\"stepType\"\:\s\"(?<steptype>.*?)\""
| rex field=_raw "\"flowname\"\:\s\"(?<flowname>.*?)\""
| rex field=_raw "INFO ((?<infotime>\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2},\d{3}))" |stats latest(eval(if(steptype="EndNBflow",max(infotime),0))) AS endNBflow latest(eval(if(steptype="Deserialized payload",infotime,0))) AS endPayLoad dc(steptype) as unique_steptypes by appid|where unique_steptypes >= 16 |eval diff=endNBflow-endPayLoad

My earlier code included- 

| rex field=_raw "INFO  ((?<infotime>\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2},\d{3}))"

| stats max(infotime) as maxinfotime, min(infotime) as mininfotime,count(eval(match(_raw, "error"))) as error_count, dc(steptype) as unique_steptypes by appid

| where error_count = 0

| eval maxtime=strptime(maxinfotime,"%Y-%m-%d %H:%M:%S,%3N") | eval mintime=strptime(mininfotime,"%Y-%m-%d %H:%M:%S,%3N") |  eval TimeDiff=maxtime-mintime | eval TimeDiff_formated = strftime(TimeDiff,"%H:%M:%S,%3N")| where unique_steptypes >= 16|sort steptype

| table appid, mininfotime, maxinfotime, mintime, maxtime, TimeDiff_formated, unique_steptypes, flowname

I am unable to club these two and get the expected output. 

[slearntrain · Accepted solution]

Alright!!! I found the answer to this question- Modified the below query by changing the time formats of the new fields and then pulling out the difference --

index="abc" sourcetype=openshift_logs openshift_namespace="qaenv" "a9ecdae5-45t6-abcd*"
| rex field=_raw "\"Application-ID\"\:\s\"(?<appid>.*?)\""
| rex field=_raw "\"stepType\"\:\s\"(?<steptype>.*?)\""
| rex field=_raw "\"flowname\"\:\s\"(?<flowname>.*?)\""
| rex field=_raw "INFO ((?<infotime>\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2},\d{3}))" |stats latest(eval(if(steptype="EndNBflow",max(infotime),0))) AS endNBflow , latest(eval(if(steptype="Deserialized payload",infotime,0))) AS endPayLoad , count(eval(match(_raw,"error"))) as error_count, dc(steptype) as unique_steptypes BY appid|where unique_steptypes >= 16 AND error_count=0|eval endNBflowtime=strptime(endNBflow, "%Y-%m-%d %H:%M:%S,%3N")
| eval endPayLoadtime=strptime(endPayLoad, "%Y-%m-%d %H:%M:%S,%3N")
| eval time_difference = endNBflowtime - endPayLoadtime
| table appid,endNBflow,endPayLoad, endNBflowtime, endPayLoadtime, time_difference

Results are like this :

Appid	endNBflow	Endpayload	endNBflowtime	Endpayloadtime	responsetime
Abcd1	2024-03-04 16:10:50,007	2024-03-04 16:10:49,886	1709529050.007000	1709529049.886000	0.121000

[gcusello]

Hi @slearntrain ,

good for you, see next time!

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated 😉