topic Re: How to create new field based on regex? in Splunk Search

How to create new field based on regex?

codewarrior — Wed, 17 Apr 2024 21:03:40 GMT

I have a log stream in this format:

level=info request.elapsed=100 request.method=GET request.path=/orders/123456 request_id=2ca011b5-ad34-4f32-a95c-78e8b5b1a270 response.status=500

I have extracted the fields using regex:

| rex field=message "level=info request.elapsed=(?<duration>.*) request.method=(?<method>.*) request.path=(?<path>.*) request_id=(?<request_id>.*) response.status=(?<statusCode>.*)"

I want to manually build a new field called route based on the extracted field path. For example, for "path=/order/123456", I want to create new field "route=/order/{orderID}", so I can grouping by route not by path, the path contains real parameter which I cannot group on path.

How can I achieve this? Thanks.

Re: How to create new field based on regex?

scelikok — Wed, 17 Apr 2024 22:32:51 GMT

Hi @codewarrior,

If I got it correct, your need is extract a new field named "route" and it will contain the value after "orders/".

You can capture it in your rex command, please try below;

level=info request.elapsed=(?<duration>.*) request.method=(?<method>.*) request.path=(?<path>.+orders\/(?<route>.+)) request_id=(?<request_id>.*) response.status=(?<statusCode>.*)

Re: How to create new field based on regex?

codewarrior — Thu, 18 Apr 2024 21:13:46 GMT

Thanks @scelikok. No I don't just want the orderID. But I want to manually create the RESTful API routing pattern.

for "path=/order/123456", "route=/order/{orderID}", basically I am trying to use regex to replace the value and create a new field in this way:

if value matches \/order\/\d{12}, then convert to /order/{orderID}

I have other examples like:

path=/user/jason@sample.com/orders

route=/user/{userID}/orders

Re: How to create new field based on regex?

yuanliu — Wed, 17 Apr 2024 23:44:56 GMT

I do not see why you needed to do that extra extraction because Splunk should have given you a field named "request_path" already. (See emulation below.) All you need to do is to assign a new field based on match.

| eval route = if(match(request_path, "^/orders/\d+"), "/order/{orderID}", null())

The sample data should give you something like

level	request_elapsed	request_id	request_method	request_path	response_status	route
info	100	2ca011b5-ad34-4f32-a95c-78e8b5b1a270	GET	/orders/123456	500	/order/{orderID}

Is this what you wanted?

Here is a data emulation you can play with and compare with real data.

| makeresults | eval _raw = "level=info request.elapsed=100 request.method=GET request.path=/orders/123456 request_id=2ca011b5-ad34-4f32-a95c-78e8b5b1a270 response.status=500" | extract ``` data emulation above ```

Of course, if for unknown reasons Splunk doesn't give you request_path, simply add an extract command and skip all the rex which is expensive.

Re: How to create new field based on regex?

codewarrior — Thu, 18 Apr 2024 21:13:11 GMT

Thanks @yuanliu, I explained why I couldn't use path directly, because it contains actual parameters.

For example, for the route /orders/{orderID}, the path could be:

/orders/123456

/orders/213123

/orders/435534

I want to analyze, for example, count of failed requests, or percentiles of call duration on this particular API route /orders/{orderID}.

Of course I can modify my service code to print the route pattern in log, but that is another way, i need to deploy new code to production environment.

Re: How to create new field based on regex?

yuanliu — Fri, 19 Apr 2024 04:25:57 GMT

Have you tried my previous code?

| eval route = if(match(request_path, "^/orders/\d+"), "/order/{orderID}", null())

This does exactly what you ask: create a new field named route that has a fixed pattern "/order/{orderID}". Is there anything wrong with this?

In fact, because you really only care about first segment of the path - that fixed string "{orderID}" is just a decoration, the command could be simplified to slightly less expensive

| eval route = "/" . mvindex(split(request_path, "/"), 1) . "/{orderID}"

You can do whatever analysis against this field.