universal rex for extract intersting fields like (url, uri, user, email, ip, etc)

[indeed_2000]

Hi

is there any universal or general rex to extract every known intersting fields like  (url, uri, user, email, ip, etc) from logs?

Thanks

[PickleRick]

Hey, if you know a way to universally extract fields from any log, give us a hint.

But seriously - there are so many log types out there that there's no possibility to parse them all the same way.

How would you want to parse let's say CSV files, json structures, CEF events and multiline events from a tk102 gps logger?

It's like asking if you know a universal way to talk to anyone, regardless of what language he speaks.

That's simply not possible.

[indeed_2000]

simple log file (one event per line no json no csv just simple and unistructural one)

Here is the pattern:

TIMESTAMP    LOGLEVEL     SOMEMESSAGE

need to whereever find (url, uri, user, email, ip, etc) extract them.

Thanks,

[gcusello]

Hi @indeed_2000,

I need to have some sample (three or four events) of your logs to create a regex for field extraction.

Ciao.

Giuseppe

[PickleRick]

If you have chunks of data that are easily distinguishable, you can make regex-based extractions that will find them wherever in the event they are. Mind you however, that not many types of data chunks like that can be reliably specified (so that they cover all possiblities and do not capture false positives).

[gcusello]

Hi @indeed_2000,

Splunk by itself recognize the pairs field=value.

Then, if you installe some Add-On for the technologies you're monitoring (e.g. Splunk_TA_Windows for Windows logs), you already have many field extractions just ready for you.

At the end, you have the possibility to create your own field extractions using regexes, but they are custom and specified for the logs you're analyzing.

In addition there's the regex101.com site that's very helpful for your regexes extractions.

If you need an help for your logs, please share them and surely you'll have a very quick help.

Bye.

Giuseppe