Splunk Search

How to create new field based on regex?

codewarrior
Loves-to-Learn Everything

I have a log stream in this format:

level=info request.elapsed=100 request.method=GET request.path=/orders/123456 request_id=2ca011b5-ad34-4f32-a95c-78e8b5b1a270 response.status=500

I have extracted the fields using regex:

| rex field=message "level=info request.elapsed=(?<duration>.*) request.method=(?<method>.*) request.path=(?<path>.*) request_id=(?<request_id>.*) response.status=(?<statusCode>.*)"

I want to manually build a new field called route based on the extracted field path. For example, for "path=/order/123456", I want to create new field "route=/order/{orderID}", so I can grouping by route not by path, the path contains real parameter which I cannot group on path. 

How can I achieve this? Thanks.

Labels (2)
0 Karma

yuanliu
SplunkTrust
SplunkTrust

I do not see why you needed to do that extra extraction because Splunk should have given you a field named "request_path" already. (See emulation below.)  All you need to do is to assign a new field based on match.

 

| eval route = if(match(request_path, "^/orders/\d+"), "/order/{orderID}", null())

 

The sample data should give you something like

levelrequest_elapsedrequest_idrequest_methodrequest_pathresponse_statusroute
info1002ca011b5-ad34-4f32-a95c-78e8b5b1a270GET/orders/123456500/order/{orderID}

Is this what you wanted?

Here is a data emulation you can play with and compare with real data.

 

| makeresults
| eval _raw = "level=info request.elapsed=100 request.method=GET request.path=/orders/123456 request_id=2ca011b5-ad34-4f32-a95c-78e8b5b1a270 response.status=500"
| extract
``` data emulation above ```

 

Of course, if for unknown reasons Splunk doesn't give you request_path, simply add an extract command and skip all the rex which is expensive.

Tags (1)
0 Karma

codewarrior
Loves-to-Learn Everything

Thanks @yuanliu, I explained why I couldn't use path directly, because it contains actual parameters. 

For example, for the route /orders/{orderID}, the path could be:

/orders/123456

/orders/213123

/orders/435534

I want to analyze, for example, count of failed requests, or percentiles of call duration on this particular API route /orders/{orderID}.  

Of course I can modify my service code to print the route pattern in log, but that is another way, i need to deploy new code to production environment. 

0 Karma

yuanliu
SplunkTrust
SplunkTrust

Have you tried my previous code?

| eval route = if(match(request_path, "^/orders/\d+"), "/order/{orderID}", null())

This does exactly what you ask: create a new field named route that has a fixed pattern "/order/{orderID}".  Is there anything wrong with this?

In fact, because you really only care about first segment of the path - that fixed string "{orderID}" is just a decoration, the command could be simplified to slightly less expensive

| eval route = "/" . mvindex(split(request_path, "/"), 1) . "/{orderID}"

You can do whatever analysis against this field.

0 Karma

scelikok
SplunkTrust
SplunkTrust

Hi @codewarrior,

If I got it correct, your need is extract a new field named "route" and it will contain the value after "orders/". 

You can capture it in your rex command, please try below;

level=info request.elapsed=(?<duration>.*) request.method=(?<method>.*) request.path=(?<path>.+orders\/(?<route>.+)) request_id=(?<request_id>.*) response.status=(?<statusCode>.*)

 

If this reply helps you an upvote and "Accept as Solution" is appreciated.
0 Karma

codewarrior
Loves-to-Learn Everything

Thanks @scelikok. No I don't just want the orderID. But I want to manually create the RESTful API routing pattern.

for "path=/order/123456",  "route=/order/{orderID}", basically I am trying to use regex to replace the value and create a new field in this way:

if value matches \/order\/\d{12}, then convert to /order/{orderID}

I have other examples like:

path=/user/jason@sample.com/orders

route=/user/{userID}/orders

 

 

0 Karma
Get Updates on the Splunk Community!

Optimize Cloud Monitoring

  TECH TALKS Optimize Cloud Monitoring Tuesday, August 13, 2024  |  11:00AM–12:00PM PST   Register to ...

What's New in Splunk Cloud Platform 9.2.2403?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.2.2403! Analysts can ...

Stay Connected: Your Guide to July and August Tech Talks, Office Hours, and Webinars!

Dive into our sizzling summer lineup for July and August Community Office Hours and Tech Talks. Scroll down to ...