Splunk Search

How to rex?

mikeyty07
Communicator

I have 2 kind of logs where there are two types of uri which i want to rex into different fields

{logType=DOWNSTREAM_RESPONSE, requestUri=https://google.come.com:8000/google/api/updateapi?&lo=en_US&sc=RT, duration=22, requestId=znXdSxbJQw6iVTtEeykZVA, globalTrackingId=null, requestTrackingId=null, request={body={"a":{"b":{"country":"US", }}}, method=POST, requestUri=https://google.come.com:443/google/api/updateapi?&lo=en_US&sc=RT}, response=(200 OK, { "body="{} }, "headers="{}, "statusCode=OK", statusCodeValue=200}")"}

{logType=DOWNSTREAM_RESPONSE, requestUri=https://google.come.com:8000/google/api/deleteapi, duration=33, requestId=asdasd, globalTrackingId=null, requestTrackingId=null, request={body={"a":{"b":{"country":"US", }}}, method=POST, requestUri=https://google.come.com:443/google/api/updateapi?&lo=en_US&sc=RT}, response=(200 OK, { "body="{} }, "headers="{}, "statusCode=OK", statusCodeValue=200}")"}

http= https

URL= google.come.com:8000

service = /google

api= /api/updateapi
api= /api/deleteapi

params= ?&lo=en_US&sc=RT

is there a way to regex this?

 

Labels (4)
1 Solution

Tom_Lundie
Contributor

Hi,

Yes you can extract those fields using | rex.

Try this:

| rex max_match=0 "(?<http>https?)\:\/\/(?<URL>[^\/]*)(?<service>/[^\/]*)(?<api>[^\?\,\}]*)(?:\?(?<params>[^\,\}]*))?"

This will extract all URLs in the _raw event into those fields that you suggested. If you want to run this on a specific field then you could add the field=<<field>> argument to the | rex command.

View solution in original post

inventsekar
SplunkTrust
SplunkTrust

Hi @mikeyty07 ... its pretty simple actually.. check this out.. 

 

source="two-logs.txt" host="laptop" sourcetype="twologs" 
| rex field=_raw "requestUri\=(?P<status>\w+)\:\/\/(?P<URL>\w+\.\w+\.\w+\:\d+)(?P<service>\/\w+)(?P<api>\/\w+\/\w+)[\,|\?](?P<params>\S+)\,"  
| table status URL service api params

 

check the sample run:

rex-two-logs.jpg

 

Tom_Lundie
Contributor

Hi,

Yes you can extract those fields using | rex.

Try this:

| rex max_match=0 "(?<http>https?)\:\/\/(?<URL>[^\/]*)(?<service>/[^\/]*)(?<api>[^\?\,\}]*)(?:\?(?<params>[^\,\}]*))?"

This will extract all URLs in the _raw event into those fields that you suggested. If you want to run this on a specific field then you could add the field=<<field>> argument to the | rex command.

mikeyty07
Communicator

if i have to add the duration in same query of rex what would it be?

0 Karma

Tom_Lundie
Contributor

Looking at your sample logs, duration is a separate field and it only seems to be applied in context of one of your URLs so I wouldn't extract it within the same | rex query.

Best-practise here is to set-up search-time field extractions for these fields. Read more here. The general idea is to make your fields extract automatically so that you don't have to run these rex commands for every single-use case that pertains to these logs.

That being said, if you're sure that you want to get this extracted via rex, use a separate command, the following regex will work:

| rex "duration=(?<duration>\d+)"

 

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...