topic Re: Is there an alternative to using regex in my search for better performance? in Splunk Search

Is there an alternative to using regex in my search for better performance?

sieutruc — Thu, 16 Jun 2016 14:50:59 GMT

hello,

After reading some answers, I see that if I use regex for searching events corresponding to a pattern, it will take a lot of time as Splunk reads all events from disk.

For example: I use index=X email="test@*", it will be so much faster than index=X | regex email="test@.*".

So my question is beside the * , can I use another regex term in the default search without using regex that provides the same performance as original search.

For ex:
index=X email="test@[a-z]+.com" ?
index=X email="test@[0-9]*.com" ?

Re: Is there an alternative to using regex in my search for better performance?

somesoni2 — Thu, 16 Jun 2016 16:24:40 GMT

Regular expressions are not supported in base search (only wild card */ asterisk ). I would suggest to add some filters in the base search using wildcard and then use regex to do to the point filter (hybrid of both type of filter).

Re: Is there an alternative to using regex in my search for better performance?

jdunlea — Thu, 16 Jun 2016 16:30:29 GMT

From quickly scanning through some documentation, it seems that "rex" is actually a "distributed streaming" command which means it can be run on the indexer itself so you don't have to worry about innefficiencies with map-reduce.

However, to better structure your search you can provide all the "known search tokens" to your search and you could do something like this:

index=x test @ | regex email="test@.*"

What this does is it passes the known "search tokens" of "test" and "@" as search tokens to the indexer which allows the indexer to pull out only events with those two tokens anywhere in the event. THEN the "rex" will do the specific pattern match. I dont think doing the rex on it's own will allow the indexer to search for events where ONLY "test" and "@" are present. It will have to search ALL events first.

So the search above reduces inefficiencies because it puts everything that you know you need before the first pipe and then allows the rex to do the pattern matching afterwards.

Re: Is there an alternative to using regex in my search for better performance?

sieutruc — Fri, 17 Jun 2016 10:48:29 GMT

your answer is just right for some specific cases. if i search for "email="test.*hello@.*", the search with the tokens like "test hello @" will return nothing.

Re: Is there an alternative to using regex in my search for better performance?

sieutruc — Fri, 17 Jun 2016 10:52:42 GMT

thank for your reply, only * is supported in base search ( cannot use ?, [0-9], or [a-z] ), is it right ?
I ask this type of question because i did not where the doc of splunk mentions all regular expressions that could be used in base search.

Re: Is there an alternative to using regex in my search for better performance?

jdunlea — Fri, 17 Jun 2016 13:12:49 GMT

Well, that is only true if "test" and "hello" are not individual tokens.

I.E. If I search as follows:

index=X test hello @ | rex email="test.*hello@.*"

This will NOT return any results IF the data you are looking for is something like

"testworldhello@something.com"

This is because you cannot search for "test" or "hello" on their own if they are just a part of a larger token (testworldhello).

The search above WILL return results if the data looks like:

"test.world-hello@something.com"

The main point I am trying to make is that to create better search efficiency you can provide as many actual tokens as you can, up front. Tokens are separated by things like dots, dashes, slashes, etc.

To see how tokens are identified and separated in Splunk you can research segmenters.conf which shows you how Splunk breaks out tokens in any event.

Re: Is there an alternative to using regex in my search for better performance?

jmallorquin — Fri, 17 Jun 2016 13:16:24 GMT

Hi,

Have you try to extract the fields of the email like @

Then if you make a search using these fields it should be faster like

index=aaaa field1=test field2=google.com

Hope i help you

Re: Is there an alternative to using regex in my search for better performance?

somesoni2 — Fri, 17 Jun 2016 13:39:24 GMT

The base search provides all the options a "| search" command provides (actually they are the same, it's hidden in base search). It basically uses logical expression (not regular expressions). See more info here
http://docs.splunk.com/Documentation/Splunk/6.4.1/SearchReference/Search#Usage

Re: Is there an alternative to using regex in my search for better performance?

sieutruc — Fri, 17 Jun 2016 14:21:02 GMT

Thank you for your information, now i know only * is supported. I hope splunk would support more wild card in the future version 🙂 , for ex: "?" or "|".

Re: Is there an alternative to using regex in my search for better performance?

sieutruc — Fri, 17 Jun 2016 14:27:04 GMT

it does not actually respond to my question because if field 1 contains un regular expression that is not "*" wild card, you have to use regex command and ... splunk reads all events for the comparision. I think the temporary solution maybe use the hydrid solution like the answer of @somesoni2 above

Re: Is there an alternative to using regex in my search for better performance?

sieutruc — Fri, 17 Jun 2016 14:29:17 GMT

i got your point, but that's the reason i asked this question, i want to know if splunk supports more than the asterisk wild card in the base search. Thank you in anyway.

Re: Is there an alternative to using regex in my search for better performance?

jdunlea — Fri, 17 Jun 2016 14:32:06 GMT

Ah I see. Looks like you got your answer above! Good luck! 🙂

Re: Is there an alternative to using regex in my search for better performance?

jmallorquin — Mon, 20 Jun 2016 09:04:48 GMT

Ok regards 🙂