rex for dynamic parttern

[phamxuantung]

Hi, I have a raw log with structure like this:

TIME|FROM|TO|URL|ERROR|STATUS|ALERT

Example:

Wed Jan 6 15:10:01 2021|Department A|Department B|www.abc.com|0|Connected|Call Department C

I want to use rex to dissect them to their own fields like TIME, FROM, TO, etc... But the thing is every fields in the log are dynamic, meaning they don't always stay the same. I pretty new to regular expression and splunk so I don't know how to operate this.

Thank you in advance.

[ITWhisperer]

It depends what you mean by dynamic. Does this work for you?

| rex "(?P<TIME>[^\|]*)\|(?P<FROM>[^\|]*)\|(?P<TO>[^\|]*)\|(?P<URL>[^\|]*)\|(?P<ERROR>[^\|]*)\|(?P<STATUS>[^\|]*)\|(?P<ALERT>.*)"

[gcusello]

Hi @phamxuantung,

let me understand: the problem is that the order of the fields is changing or that some field sometimes is missing?

is it possible to define some additional rules for the values or the format of the fields (e.g.: STATUS can be only "Connected" or "disconnected"; FROM and TO starts always with "Department"; etc...)?

If you can define some rules it's possible to read the fields, otherwise it's difficoult.

Ciao.

Giuseppe

[phamxuantung]

Hi @gcusello,

The problem is the values of the field are changing, with varied length, The rule is the order and they separate with |. Detail of each field would be:

TIME: The time of the events.

FROM|TO: Name of the Gateway, you can think of them like people name, and they have a set of 10 values or so.

URL: The URL of the connected service - The only field that have a static start pattern ("http:"), but the content varied from a web address that end with ".com" or an IP address like 192.168.1.1.

ERROR: Between 0 and 1 whereas 0 is Connected and 1 is Error.

STATUS: "Connecting to" above URL...connected/error.

ALERT: always stay the same, "Call IT", but I would like to also take it out in case there might be more in the future.

[gcusello · Accepted solution]

Hi @phamxuantung,

if the order is always the same, it's easy:

| rex "^(?<TIME>[^\|]+)\|(?<FROM>[^\|]+)\|(?<TO>[^\|]+)\|(?<URL>[^\|]+)\|(?<ERROR>[^\|]+)\|(?<STATUS>[^\|]+)\|(?<ALERT>.+)"

You can test the regex at https://regex101.com/r/6pQnRw/1.

Ciao.

Giuseppe

[gcusello]

Hi @phamxuantung,

good for you.

Ciao and happy splunking.

Giuseppe

P.S.: Karma Points are appreciated by alla contributors 😉