topic Re: Help with REX in Splunk Search

Help with REX

1234testtest — Wed, 13 Feb 2013 09:04:21 GMT

rex "(?i)\(ms\):(?P<duration>.+)"

Query:

sourcetype="mylog" | rex "(?i)\(ms\):(?P<duration>.+)" |  eval epochtime=_time |   sort epochtime | table epochtime,duration

I am unable to figure out what is happening within the rex statement above.

(ms):5.276144

(Data in log file comes in this format).
I can understand that we're trying to catch "(ms):" in the rex command through "\(ms\):"
but am not able to figure out the rest of it.
Kindly help.

This command is extremely slow in performance - would request what changes could be done to better the performance.

Re: Help with REX

Ayn — Wed, 13 Feb 2013 09:20:24 GMT

So what you're saying is you wrote a query that you don't understand what it does? Is something not working, or is it working but you don't understand why?

Also I'm not aware of any particular performance issues with rex - do you mean that the performance of rex itself is poor, or do you mean that the whole query with or without rex is slow?

Re: Help with REX

1234testtest — Wed, 13 Feb 2013 09:49:05 GMT

To clarify - query was written by someone else who left the firm 😞
1. I didnt understand what the query is doing - I'm just getting the output of time and duration (how duration is extracted and what does + symbol denote - is it adding anything/ and overall what is the segment right to rex doing. Unable to get it.
2. Whole query is slow - assumption is that rex and segment next to rex are taking too much time by splunk to evalute.

Re: Help with REX

jonuwz — Wed, 13 Feb 2013 13:04:26 GMT

"(?i)\(ms\):(?P<duration>.+)"

(?i)             = case insensitive  
\(ms\):          = match (ms): literally
(?P<duration>.+) = match 1 or more characters and make this available in the 'duration' field

It is extremely unlikely that the rex is the bottleneck. For something that simple, the regex engine would process tens of thousands of lines a second, and probably a lot more.

Put your search in the search bar.
Let it run, then press the Job Inspector button (the blue one with "i" on it).

You'll see what parts of the search take longest.

command.rex will be listed somewhere - its likely that its a very short bar.

Re: Help with REX

1234testtest — Thu, 14 Feb 2013 12:16:12 GMT

The following are the timings shown - looks like command.search also is taking time. There are 26,128 matching events.

1.249 command.search
0.525 command.search.typer

0.436 command.search.rawdata
0.355 command.sort
1.518 dispatch.fetch 13 - -
2.799 dispatch.preview
1.514 dispatch.stream.local
1.779 dispatch.timeline

1.173 dispatch.tmpevents
0.546 startup.handoff