Splunk Search

rex for dynamic parttern

phamxuantung
Path Finder

Hi, I have a raw log with structure like this:

 

TIME|FROM|TO|URL|ERROR|STATUS|ALERT

 

Example:

Wed Jan 6 15:10:01 2021|Department A|Department B|www.abc.com|0|Connected|Call Department C

I want to use rex to dissect them to their own fields like TIME, FROM, TO, etc... But the thing is every fields in the log are dynamic, meaning they don't always stay the same. I pretty new to regular expression and splunk so I don't know how to operate this.

Thank you in advance.

Labels (3)
0 Karma
1 Solution

gcusello
SplunkTrust
SplunkTrust

Hi @phamxuantung,

if the order is always the same, it's easy:

 

| rex "^(?<TIME>[^\|]+)\|(?<FROM>[^\|]+)\|(?<TO>[^\|]+)\|(?<URL>[^\|]+)\|(?<ERROR>[^\|]+)\|(?<STATUS>[^\|]+)\|(?<ALERT>.+)"

 

You can test the regex at https://regex101.com/r/6pQnRw/1.

Ciao.

Giuseppe

View solution in original post

0 Karma

ITWhisperer
Ultra Champion

It depends what you mean by dynamic. Does this work for you?

| rex "(?P<TIME>[^\|]*)\|(?P<FROM>[^\|]*)\|(?P<TO>[^\|]*)\|(?P<URL>[^\|]*)\|(?P<ERROR>[^\|]*)\|(?P<STATUS>[^\|]*)\|(?P<ALERT>.*)"
0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @phamxuantung,

let me understand: the problem is that the order of the fields is changing or that some field sometimes is missing?

is it possible to define some additional rules for the values or the format of the fields (e.g.: STATUS can be only "Connected" or "disconnected"; FROM and TO starts always with "Department"; etc...)?

If you can define some rules it's possible to read the fields, otherwise it's difficoult.

Ciao.

Giuseppe

0 Karma

phamxuantung
Path Finder

Hi @gcusello,

The problem is the values of the field are changing, with varied length, The rule is the order and they separate with |. Detail of each field would be:

TIME: The time of the events.

FROM|TO: Name of the Gateway, you can think of them like people name, and they have a set of 10 values or so.

URL: The URL of the connected service - The only field that have a static start pattern ("http:"), but the content varied from a web address that end with ".com" or an IP address like 192.168.1.1.

ERROR: Between 0 and 1 whereas 0 is Connected and 1 is Error.

STATUS: "Connecting to" above URL...connected/error.

ALERT: always stay the same, "Call IT", but I would like to also take it out in case there might be more in the future.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @phamxuantung,

if the order is always the same, it's easy:

 

| rex "^(?<TIME>[^\|]+)\|(?<FROM>[^\|]+)\|(?<TO>[^\|]+)\|(?<URL>[^\|]+)\|(?<ERROR>[^\|]+)\|(?<STATUS>[^\|]+)\|(?<ALERT>.+)"

 

You can test the regex at https://regex101.com/r/6pQnRw/1.

Ciao.

Giuseppe

View solution in original post

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @phamxuantung,

good for you.

Ciao and happy splunking.

Giuseppe

P.S.: Karma Points are appreciated by alla contributors 😉

0 Karma
Take the 2021 Splunk Career Survey

Help us learn about how Splunk has
impacted your career by taking the 2021 Splunk Career Survey.

Earn $50 in Amazon cash!