Splunk Search

How to rex?

mikeyty07
Communicator

I have 2 kind of logs where there are two types of uri which i want to rex into different fields

{logType=DOWNSTREAM_RESPONSE, requestUri=https://google.come.com:8000/google/api/updateapi?&lo=en_US&sc=RT, duration=22, requestId=znXdSxbJQw6iVTtEeykZVA, globalTrackingId=null, requestTrackingId=null, request={body={"a":{"b":{"country":"US", }}}, method=POST, requestUri=https://google.come.com:443/google/api/updateapi?&lo=en_US&sc=RT}, response=(200 OK, { "body="{} }, "headers="{}, "statusCode=OK", statusCodeValue=200}")"}

{logType=DOWNSTREAM_RESPONSE, requestUri=https://google.come.com:8000/google/api/deleteapi, duration=33, requestId=asdasd, globalTrackingId=null, requestTrackingId=null, request={body={"a":{"b":{"country":"US", }}}, method=POST, requestUri=https://google.come.com:443/google/api/updateapi?&lo=en_US&sc=RT}, response=(200 OK, { "body="{} }, "headers="{}, "statusCode=OK", statusCodeValue=200}")"}

http= https

URL= google.come.com:8000

service = /google

api= /api/updateapi
api= /api/deleteapi

params= ?&lo=en_US&sc=RT

is there a way to regex this?

 

Labels (4)
1 Solution

Tom_Lundie
Contributor

Hi,

Yes you can extract those fields using | rex.

Try this:

| rex max_match=0 "(?<http>https?)\:\/\/(?<URL>[^\/]*)(?<service>/[^\/]*)(?<api>[^\?\,\}]*)(?:\?(?<params>[^\,\}]*))?"

This will extract all URLs in the _raw event into those fields that you suggested. If you want to run this on a specific field then you could add the field=<<field>> argument to the | rex command.

View solution in original post

inventsekar
SplunkTrust
SplunkTrust

Hi @mikeyty07 ... its pretty simple actually.. check this out.. 

 

source="two-logs.txt" host="laptop" sourcetype="twologs" 
| rex field=_raw "requestUri\=(?P<status>\w+)\:\/\/(?P<URL>\w+\.\w+\.\w+\:\d+)(?P<service>\/\w+)(?P<api>\/\w+\/\w+)[\,|\?](?P<params>\S+)\,"  
| table status URL service api params

 

check the sample run:

rex-two-logs.jpg

 

thanks and best regards,
Sekar

PS - If this or any post helped you in any way, pls consider upvoting, thanks for reading !

Tom_Lundie
Contributor

Hi,

Yes you can extract those fields using | rex.

Try this:

| rex max_match=0 "(?<http>https?)\:\/\/(?<URL>[^\/]*)(?<service>/[^\/]*)(?<api>[^\?\,\}]*)(?:\?(?<params>[^\,\}]*))?"

This will extract all URLs in the _raw event into those fields that you suggested. If you want to run this on a specific field then you could add the field=<<field>> argument to the | rex command.

mikeyty07
Communicator

if i have to add the duration in same query of rex what would it be?

0 Karma

Tom_Lundie
Contributor

Looking at your sample logs, duration is a separate field and it only seems to be applied in context of one of your URLs so I wouldn't extract it within the same | rex query.

Best-practise here is to set-up search-time field extractions for these fields. Read more here. The general idea is to make your fields extract automatically so that you don't have to run these rex commands for every single-use case that pertains to these logs.

That being said, if you're sure that you want to get this extracted via rex, use a separate command, the following regex will work:

| rex "duration=(?<duration>\d+)"

 

0 Karma
Get Updates on the Splunk Community!

What You Read The Most: Splunk Lantern’s Most Popular Articles!

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Announcing the General Availability of Splunk Enterprise Security 8.1!

We are pleased to announce the general availability of Splunk Enterprise Security 8.1. Splunk becomes the only ...

Developer Spotlight with William Searle

The Splunk Guy: A Developer’s Path from Web to Cloud William is a Splunk Professional Services Consultant with ...