Rex for https status code, response time and url list

[ravir_jbp]

I have below HTTP events where in I am trying to extract status code, response time and URL. I am using the following rex query below. This query works fine to find error 200, 400 and 500. But not for 30* errors. If you refer the below 302 events its has event like (302 - ). other status has like(200 87909) (400 568). Can you help me with the expression that is missing so that it extract for all the codes. I verfied it in "https://regex101.com/r/bVp3gz/1" as well.

===============

HTTP\/1.1\"\s(?\d+)\s(?\d+)\s"(?[^\"]*)"

==========================

11.111.111.1 [04/Mar/2021:09:05:40 -0600] 1061614 "GET /merced/content/frag/breeze/bootstrap/fonts/icomoon.ttf?az1hj2 HTTP/1.1" 200 95364 "https://sfdfdsfsd-sfsdfasf.topms.com/mxxx/treports/prepackaged/O-Rx_Agent_MyUnacknowledgedCoachingSe..." "Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko" - B6613D2EAB90BFAB32BD90BA61E8280E.app1

11.111.111.11 [04/Mar/2021:09:36:41 -0600] 169017 "GET /delegate/forwarderServlet/process.do?url=%2Fmerced%2Fdashboards%2FO-Rx_Agent_HomePage_Dash%3Flang%3Den_US&appid=xxx HTTP/1.1" 302 - "https://sfdfdsfsd-sfsdfasf.topms.com/group/npm/o-rx_agent_homepage_dash" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.190 Safari/537.36" - C9A3D61145B15DEFCD8BD71736242EA8.tomcat2

11.111.111.1 [04/Mar/2021:08:35:20 -0600] 17580 "GET /merced/populate?assistant=person&query=jomalyn%2520mallari&policyName=%2Fcom%2Fmerced%2Fmodels%2Femployee%2Fpolicies%2FCoachingWritePolicy&fieldName=EEDRFE HTTP/1.1" 500 977 "https://sfdfdsfsd-sfsdfasf.topms.com/mxxx/forms/BPLCoachingSessionForm?lang=en_US" "Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko" - ABEAA3DED8CBC4163D544C986AA455DA.app4

11.111.111.1 [04/Mar/2021:10:00:27 -0600] 0 "GET /nice-documentation/javascripts/MercedHelpLib.js?browserId=other&minifierType=js&languageId=en_US&b=0000&t=1612576281967 HTTP/1.1" 404 1083 "https://sfdfdsfsd-sfsdfasf.topms.com/group/xxxx/o-rx_agent_homepage_dash" "Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko" - A8B66CD6680C8747D4C878CAAE64B1D7.tomcat1

[scelikok]

Sorry about missing escape and quotes;

| rex "HTTP\/1.1\"\s(?<status>\d+)\s(?<restime>\d+|-)\s\"(?<url>[^\"]*)"

[scelikok]

Hi @ravir_jbp,

Please try using below, response time is not decimal on 302 messages,

rex HTTP\/1.1\"\s(?<status>\d+)\s(?<restime>\d+|-)\s"(?<url>[^\"]*)"

[ravir_jbp]

I am getting this error:

Error in 'SearchParser': Missing a search command before '-'. Error at position '80' of search query 'search index=cba_XXXXX_XXXX |rex HTTP\/1.1\"\s(...{snipped} {errorcontext = stime>\d+|-)\s"(?<url}'.

[ITWhisperer]

Looks like you are missing a backslash before the double quote at the start of the URL

[manjunathmeti]

Try this:

| rex "HTTP\/1.1\"\s(?<status_code>\d+)\s(?<response_time>(?:\d+|\-))\s\"(?<URL>[^\"]*)\""

[ravir_jbp]

Hi,

Sorry I posted the wrong query that I wrote. The one I am using is:

Rex HTTP\/1.1\"\s(?<status>\d+)\s(?<restime>\d+)\s"(?<url>[^\"]*)"

This one is not working for  below 302 event. For other its owrking.

11.111.111.11 [04/Mar/2021:09:36:41 -0600] 169017 "GET /delegate/forwarderServlet/process.do?url=%2Fmerced%2Fdashboards%2FO-Rx_Agent_HomePage_Dash%3Flang%3Den_US&appid=xxx HTTP/1.1" 302 - "https://sfdfdsfsd-sfsdfasf.topms.com/group/npm/o-rx_agent_homepage_dash" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.190 Safari/537.36" - C9A3D61145B15DEFCD8BD71736242EA8.tomcat2

[manjunathmeti]

Did you try the regex I posted? It works for logs containing 302 - also.

[ravir_jbp]

Hello Manjunath,

Yes it worked for my search. I was able to get the diffeent status codes.

index=XX-XX_XX |rex "HTTP\/1.1\"\s(?<status_code>\d+)\s(?<response_time>(?:\d+|\-))\s\"(?<URL>[^\"]*)\"" | stats count by status_code

[ravir_jbp]

Thank you Manjunath.

Need your help on the requirement. Curently I am using the rex query you have provided to search for different status successfully. Now I am trying to get the top 10 URL which were accessed by user but got error messages. May be with 500 or 400 or 300 status code URLs. 

index=XXXXXXXX |rex "HTTP\/1.1\"\s(?<status_code>\d+)\s(?<response_time>(?:\d+|\-))\s\"(?<URL>[^\"]*)\"" |

[manjunathmeti]

Use top command:

| top URL

[ravir_jbp]

Hello manjunath, THat work worked. THank you for quick response. I have another question related to Rex as mentioned below

event:

=========

<Mar 8, 2021 03:16:12,638 CST> <Info> <Health> <hostname> <containername> <weblogic.GCMonitor> <<anonymous>> <> <e1d36038-3bd0-4ace-beb3-b9b78707786e-00000006> <1615194972638> <[severity-value: 64] [rid: 0:1] [partition-id: 0] [partition-name: DOMAIN] > <BEA-310002> <88% of the total memory in the server is free.>

I am also trying to use rex for following event but the previous one is not working.
I am trying to find the memory %for each "<containername>" for each "<host>". for above event, 88% is total memory for specific <containername> and <hostname>. Please help how to exact the memory % from above event.

[ravir_jbp]

Hello @manjunathmeti ,

Actually I like to extract the memory for each container for every host. One host may have multiple containers. In below example events I have four event (for two hosts) but has two containers in one host (contianer name is agent-desktop3 and avgent-desktop2). I am trying to exact in below table format:

Host name | container |  memory

<Mar 9, 2021 02:30:29,585 CST> <Info> <Health> <hostname1.host.com> <agent-desktop3> <weblogic.GCMonitor> <<anonymous>> <> <d19b10c2-4d5d-43ad-a4f5-bc0dc842a71b-00000007> <1615278629585> <[severity-value: 64] [rid: 0:1] [partition-id: 0] [partition-name: DOMAIN] > <BEA-310002> <44% of the total memory in the server is free.>

####<Mar 9, 2021 02:30:11,563 CST> <Info> <Health> <hostname1.host.com> <agent-desktop2> <weblogic.GCMonitor> <<anonymous>> <> <e1d36038-3bd0-4ace-beb3-b9b78707786e-00000006> <1615278611563> <[severity-value: 64] [rid: 0:1] [partition-id: 0] [partition-name: DOMAIN] > <BEA-310002> <58% of the total memory in the server is free.>

<Mar 9, 2021 02:30:29,585 CST> <Info> <Health> <hostname2.host.com> <agent-desktop1> <weblogic.GCMonitor> <<anonymous>> <> <d19b10c2-4d5d-43ad-a4f5-bc0dc842a71b-00000007> <1615278629585> <[severity-value: 64] [rid: 0:1] [partition-id: 0] [partition-name: DOMAIN] > <BEA-310002> <44% of the total memory in the server is free.>

####<Mar 9, 2021 02:30:11,563 CST> <Info> <Health> <hostname2.host.com> <agent-desktop2> <weblogic.GCMonitor> <<anonymous>> <> <e1d36038-3bd0-4ace-beb3-b9b78707786e-00000006> <1615278611563> <[severity-value: 64] [rid: 0:1] [partition-id: 0] [partition-name: DOMAIN] > <BEA-310002> <58% of the total memory in the server is free.>

[manjunathmeti · Accepted solution]

Try this:

| rex "\<Health\>\s\<(?<host>[^\>]+)\>\s\<(?<container>[^\>]+)\>.*\<(?<memory>\d+)\%\sof\sthe\stotal\smemory"

If this reply helps you, an upvote/like would be appreciated.

[ravir_jbp]

Hi @manjunathmeti 

IT worked!!! thank you so much for all your help. 

[manjunathmeti]

You need a different regex for this. Try this:

| rex "\<(?<memory>\d+)\%\sof\sthe\stotal\smemory"

If these replies help you, upvote/likes would be appreciated.