topic universal rex for extract intersting fields like (url, uri, user, email, ip, etc) in Splunk Search

universal rex for extract intersting fields like (url, uri, user, email, ip, etc)

indeed_2000 — Sat, 23 Oct 2021 10:08:58 GMT

is there any universal or general rex to extract every known intersting fields like (url, uri, user, email, ip, etc) from logs?

Thanks

Re: universal rex for extract intersting fields like (url, uri, user, email, ip, etc)

gcusello — Sat, 23 Oct 2021 10:14:59 GMT

Hi @indeed_2000,

Splunk by itself recognize the pairs field=value.

Then, if you installe some Add-On for the technologies you're monitoring (e.g. Splunk_TA_Windows for Windows logs), you already have many field extractions just ready for you.

At the end, you have the possibility to create your own field extractions using regexes, but they are custom and specified for the logs you're analyzing.

In addition there's the regex101.com site that's very helpful for your regexes extractions.

If you need an help for your logs, please share them and surely you'll have a very quick help.

Bye.

Giuseppe

Re: universal rex for extract intersting fields like (url, uri, user, email, ip, etc)

PickleRick — Sat, 23 Oct 2021 15:57:38 GMT

Hey, if you know a way to universally extract fields from any log, give us a hint.

But seriously - there are so many log types out there that there's no possibility to parse them all the same way.

How would you want to parse let's say CSV files, json structures, CEF events and multiline events from a tk102 gps logger?

It's like asking if you know a universal way to talk to anyone, regardless of what language he speaks.

That's simply not possible.

Re: universal rex for extract intersting fields like (url, uri, user, email, ip, etc)

indeed_2000 — Sat, 23 Oct 2021 16:04:40 GMT

simple log file (one event per line no json no csv just simple and unistructural one)

Here is the pattern:

TIMESTAMP LOGLEVEL SOMEMESSAGE

need to whereever find (url, uri, user, email, ip, etc) extract them.

Thanks,

Re: universal rex for extract intersting fields like (url, uri, user, email, ip, etc)

PickleRick — Sat, 23 Oct 2021 19:47:25 GMT

If you have chunks of data that are easily distinguishable, you can make regex-based extractions that will find them wherever in the event they are. Mind you however, that not many types of data chunks like that can be reliably specified (so that they cover all possiblities and do not capture false positives).

Re: universal rex for extract intersting fields like (url, uri, user, email, ip, etc)

gcusello — Sun, 24 Oct 2021 05:50:11 GMT

Hi @indeed_2000,

I need to have some sample (three or four events) of your logs to create a regex for field extraction.

Ciao.

Giuseppe