topic Re: Rex for https status code, response time and url list in Monitoring Splunk

Rex for https status code, response time and url list

ravir_jbp — Thu, 04 Mar 2021 16:40:08 GMT

I have below HTTP events where in I am trying to extract status code, response time and URL. I am using the following rex query below. This query works fine to find error 200, 400 and 500. But not for 30* errors. If you refer the below 302 events its has event like (302 - ). other status has like(200 87909) (400 568). Can you help me with the expression that is missing so that it extract for all the codes. I verfied it in "https://regex101.com/r/bVp3gz/1" as well.

===============

HTTP\/1.1\"\s(?\d+)\s(?\d+)\s"(?[^\"]*)"

==========================

11.111.111.1 [04/Mar/2021:09:05:40 -0600] 1061614 "GET /merced/content/frag/breeze/bootstrap/fonts/icomoon.ttf?az1hj2 HTTP/1.1" 200 95364 "https://sfdfdsfsd-sfsdfasf.topms.com/mxxx/treports/prepackaged/O-Rx_Agent_MyUnacknowledgedCoachingSessions?lang=en_US" "Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko" - B6613D2EAB90BFAB32BD90BA61E8280E.app1

11.111.111.11 [04/Mar/2021:09:36:41 -0600] 169017 "GET /delegate/forwarderServlet/process.do?url=%2Fmerced%2Fdashboards%2FO-Rx_Agent_HomePage_Dash%3Flang%3Den_US&appid=xxx HTTP/1.1" 302 - "https://sfdfdsfsd-sfsdfasf.topms.com/group/npm/o-rx_agent_homepage_dash" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.190 Safari/537.36" - C9A3D61145B15DEFCD8BD71736242EA8.tomcat2

11.111.111.1 [04/Mar/2021:08:35:20 -0600] 17580 "GET /merced/populate?assistant=person&query=jomalyn%2520mallari&policyName=%2Fcom%2Fmerced%2Fmodels%2Femployee%2Fpolicies%2FCoachingWritePolicy&fieldName=EEDRFE HTTP/1.1" 500 977 "https://sfdfdsfsd-sfsdfasf.topms.com/mxxx/forms/BPLCoachingSessionForm?lang=en_US" "Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko" - ABEAA3DED8CBC4163D544C986AA455DA.app4

11.111.111.1 [04/Mar/2021:10:00:27 -0600] 0 "GET /nice-documentation/javascripts/MercedHelpLib.js?browserId=other&minifierType=js&languageId=en_US&b=0000&t=1612576281967 HTTP/1.1" 404 1083 "https://sfdfdsfsd-sfsdfasf.topms.com/group/xxxx/o-rx_agent_homepage_dash" "Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko" - A8B66CD6680C8747D4C878CAAE64B1D7.tomcat1

Re: Rex for https status code, response time and url list

manjunathmeti — Thu, 04 Mar 2021 18:12:15 GMT

Try this:

| rex "HTTP\/1.1\"\s(?<status_code>\d+)\s(?<response_time>(?:\d+|\-))\s\"(?<URL>[^\"]*)\""

Re: Rex for https status code, response time and url list

ITWhisperer — Thu, 04 Mar 2021 18:08:45 GMT

Looks like you are missing a backslash before the double quote at the start of the URL

Re: Rex for https status code, response time and url list

ravir_jbp — Fri, 05 Mar 2021 05:42:05 GMT

Hi,

Sorry I posted the wrong query that I wrote. The one I am using is:

Rex HTTP\/1.1\"\s(?<status>\d+)\s(?<restime>\d+)\s"(?<url>[^\"]*)"

This one is not working for below 302 event. For other its owrking.

Re: Rex for https status code, response time and url list

scelikok — Fri, 05 Mar 2021 05:48:48 GMT

Hi @ravir_jbp,

Please try using below, response time is not decimal on 302 messages,

rex HTTP\/1.1\"\s(?<status>\d+)\s(?<restime>\d+|-)\s"(?<url>[^\"]*)"

Re: Rex for https status code, response time and url list

ravir_jbp — Fri, 05 Mar 2021 05:52:36 GMT

I am getting this error:

Error in 'SearchParser': Missing a search command before '-'. Error at position '80' of search query 'search index=cba_XXXXX_XXXX |rex HTTP\/1.1\"\s(...{snipped} {errorcontext = stime>\d+|-)\s"(?<url}'.

Re: Rex for https status code, response time and url list

scelikok — Fri, 05 Mar 2021 05:58:47 GMT

Sorry about missing escape and quotes;

| rex "HTTP\/1.1\"\s(?<status>\d+)\s(?<restime>\d+|-)\s\"(?<url>[^\"]*)"

Re: Rex for https status code, response time and url list

manjunathmeti — Fri, 05 Mar 2021 06:28:15 GMT

Did you try the regex I posted? It works for logs containing 302 - also.

Re: Rex for https status code, response time and url list

ravir_jbp — Sat, 06 Mar 2021 10:11:10 GMT

Hello Manjunath,

Yes it worked for my search. I was able to get the diffeent status codes.

index=XX-XX_XX |rex "HTTP\/1.1\"\s(?<status_code>\d+)\s(?<response_time>(?:\d+|\-))\s\"(?<URL>[^\"]*)\"" | stats count by status_code

Re: Rex for https status code, response time and url list

ravir_jbp — Sat, 06 Mar 2021 10:13:46 GMT

Thank you Manjunath.

Need your help on the requirement. Curently I am using the rex query you have provided to search for different status successfully. Now I am trying to get the top 10 URL which were accessed by user but got error messages. May be with 500 or 400 or 300 status code URLs.

index=XXXXXXXX |rex "HTTP\/1.1\"\s(?<status_code>\d+)\s(?<response_time>(?:\d+|\-))\s\"(?<URL>[^\"]*)\"" |

Re: Rex for https status code, response time and url list

manjunathmeti — Mon, 08 Mar 2021 06:20:20 GMT

Use top command:

| top URL

Re: Rex for https status code, response time and url list

ravir_jbp — Mon, 08 Mar 2021 09:53:36 GMT

Hello manjunath, THat work worked. THank you for quick response. I have another question related to Rex as mentioned below

event:

=========

I am also trying to use rex for following event but the previous one is not working.
I am trying to find the memory %for each "<containername>" for each "<host>". for above event, 88% is total memory for specific <containername> and <hostname>. Please help how to exact the memory % from above event.

Re: Rex for https status code, response time and url list

manjunathmeti — Tue, 09 Mar 2021 06:40:13 GMT

You need a different regex for this. Try this:

| rex "\<(?<memory>\d+)\%\sof\sthe\stotal\smemory"

If these replies help you, upvote/likes would be appreciated.

Re: Rex for https status code, response time and url list

ravir_jbp — Tue, 09 Mar 2021 08:35:35 GMT

Hello @manjunathmeti ,

Actually I like to extract the memory for each container for every host. One host may have multiple containers. In below example events I have four event (for two hosts) but has two containers in one host (contianer name is agent-desktop3 and avgent-desktop2). I am trying to exact in below table format:

Host name | container | memory

####<Mar 9, 2021 02:30:11,563 CST> <Info> <Health> <hostname1.host.com> <agent-desktop2> <weblogic.GCMonitor> <<anonymous>> <> <e1d36038-3bd0-4ace-beb3-b9b78707786e-00000006> <1615278611563> <[severity-value: 64] [rid: 0:1] [partition-id: 0] [partition-name: DOMAIN] > <BEA-310002> <58% of the total memory in the server is free.>

####<Mar 9, 2021 02:30:11,563 CST> <Info> <Health> <hostname2.host.com> <agent-desktop2> <weblogic.GCMonitor> <<anonymous>> <> <e1d36038-3bd0-4ace-beb3-b9b78707786e-00000006> <1615278611563> <[severity-value: 64] [rid: 0:1] [partition-id: 0] [partition-name: DOMAIN] > <BEA-310002> <58% of the total memory in the server is free.>

Re: Rex for https status code, response time and url list

manjunathmeti — Tue, 09 Mar 2021 08:54:37 GMT

Try this:

| rex "\<Health\>\s\<(?<host>[^\>]+)\>\s\<(?<container>[^\>]+)\>.*\<(?<memory>\d+)\%\sof\sthe\stotal\smemory"

If this reply helps you, an upvote/like would be appreciated.

Re: Rex for https status code, response time and url list

ravir_jbp — Tue, 09 Mar 2021 10:31:00 GMT

Hi @manjunathmeti

IT worked!!! thank you so much for all your help.