topic rex for dynamic parttern in Splunk Search

rex for dynamic parttern

phamxuantung — Mon, 18 Jan 2021 09:10:35 GMT

Hi, I have a raw log with structure like this:

Example:

I want to use rex to dissect them to their own fields like TIME, FROM, TO, etc... But the thing is every fields in the log are dynamic, meaning they don't always stay the same. I pretty new to regular expression and splunk so I don't know how to operate this.

Thank you in advance.

Re: rex for dynamic parttern

gcusello — Mon, 18 Jan 2021 10:16:51 GMT

Hi @phamxuantung,

let me understand: the problem is that the order of the fields is changing or that some field sometimes is missing?

is it possible to define some additional rules for the values or the format of the fields (e.g.: STATUS can be only "Connected" or "disconnected"; FROM and TO starts always with "Department"; etc...)?

If you can define some rules it's possible to read the fields, otherwise it's difficoult.

Ciao.

Giuseppe

Re: rex for dynamic parttern

ITWhisperer — Mon, 18 Jan 2021 13:49:32 GMT

It depends what you mean by dynamic. Does this work for you?

| rex "(?P<TIME>[^\|]*)\|(?P<FROM>[^\|]*)\|(?P<TO>[^\|]*)\|(?P<URL>[^\|]*)\|(?P<ERROR>[^\|]*)\|(?P<STATUS>[^\|]*)\|(?P<ALERT>.*)"

Re: rex for dynamic parttern

phamxuantung — Tue, 19 Jan 2021 02:36:22 GMT

Hi @gcusello,

The problem is the values of the field are changing, with varied length, The rule is the order and they separate with |. Detail of each field would be:

TIME: The time of the events.

FROM|TO: Name of the Gateway, you can think of them like people name, and they have a set of 10 values or so.

URL: The URL of the connected service - The only field that have a static start pattern ("http:"), but the content varied from a web address that end with ".com" or an IP address like 192.168.1.1.

ERROR: Between 0 and 1 whereas 0 is Connected and 1 is Error.

STATUS: "Connecting to" above URL...connected/error.

ALERT: always stay the same, "Call IT", but I would like to also take it out in case there might be more in the future.

Re: rex for dynamic parttern

gcusello — Tue, 19 Jan 2021 07:52:14 GMT

Hi @phamxuantung,

if the order is always the same, it's easy:

| rex "^(?<TIME>[^\|]+)\|(?<FROM>[^\|]+)\|(?<TO>[^\|]+)\|(?<URL>[^\|]+)\|(?<ERROR>[^\|]+)\|(?<STATUS>[^\|]+)\|(?<ALERT>.+)"

You can test the regex at https://regex101.com/r/6pQnRw/1.

Ciao.

Giuseppe

Re: rex for dynamic parttern

gcusello — Thu, 18 Feb 2021 09:21:43 GMT

Hi @phamxuantung,

good for you.

Ciao and happy splunking.

Giuseppe

P.S.: Karma Points are appreciated by alla contributors 😉