topic Re: testing new regex in Splunk Search

testing new regex

njathan — Fri, 30 Jul 2010 21:41:28 GMT

Before really putting my custom regex in transforms.conf, is there a quick way to test and debug it?

Re: testing new regex

Ledion_Bitincka — Fri, 30 Jul 2010 21:54:09 GMT

Use the rex command: ..... | rex "" | .....,

For example use the following regex to extract indexes.conf into the conf_file variable in the following event:

05-21-2010 17:41:51.166 INFO  IndexProcessor - indexes.conf - memPoolMB param autotuned to 512MB
... | rex " - (?<conf_file>\w+\.conf) - "| .....

Re: testing new regex

njathan — Fri, 30 Jul 2010 22:00:49 GMT

i assume you are talking about using the splunk command line, are you? how do i invoke the splunk command line? (sorry for sounding so n00by! installed splunk only yesterday 😛 )

Re: testing new regex

rroberts — Fri, 30 Jul 2010 22:15:32 GMT

Searching the internal index for regex extraction processor errors may help too. Like so...

index=_internal regexExtractionProcessor ERROR

7:59:13.931 AM

07-30-2010 07:59:13.931 ERROR regexExtractionProcessor - Cannot compile RE "((.AcctCode=\d{5}-)." for trade: missing )
host=class4.splunk.com Options| sourcetype=splunkd Options| source=/home/rick4uc/opt/splunk/var/log/splunk/splunkd.log Options

Re: testing new regex

Ledion_Bitincka — Wed, 25 Aug 2010 06:24:22 GMT

@njathan - I was talking about testing the regex using the rex search command - either from splunk's search UI or CLI

Re: testing new regex

southeringtonp — Fri, 10 Sep 2010 23:30:56 GMT

You may also want to look at Kodos (free) or RegexBuddy (commercial).

Re: testing new regex

gkanapathy — Fri, 10 Sep 2010 23:38:59 GMT

I usually use $SPLUNK_HOME/bin/pcregextest on the Splunk command line because it uses the exact regex engine as comes with Splunk. The command line is somewhat inconvenient and you do have to make sure that you quote a few characters correctly according to shell quoting rules. (Please note that you need to quote characters when using rex as well, and rex quoting rules are an even worse headache. Furthermore, bash shell quoting is well-defined and completely documented.)

Re: testing new regex

Lowell — Sat, 11 Sep 2010 00:27:07 GMT

I normally end up using Kodos for testing. The only down side is that it uses Python regular expression syntax instead of pcre-style regular expressions. The only difference I've see are: (1) You have to add a P in a capture group name, like: (?P<group>..) Of course, PCRE accepts both, so I often let the extra "P" in the final regex. And (2), and python (and therefore Kodos) doesn't support atomic grouping or possessive quantifiers (like \s++). But this generally hasn't been too much of a problem since I don't often use these advanced regex these features.

Re: testing new regex

Lowell — Sat, 11 Sep 2010 00:31:23 GMT

Good point about rex and the quoting considerations. Generally I've only hit issues when I have to match a literal " (double-quote) character, but it's important to be aware that you can't always just copy and paste between a rex command and a config file.

Re: testing new regex

tpflicke — Sat, 23 Nov 2013 23:31:26 GMT

Here's a quick way to test a regex in splunk by supplying test data via an eval, e.g. running a query like:

index=* | head 1 | eval testdata="123456a" | regex testdata="^.{5,6}(?<!p)a$" | stats count

The result is 1 if the regex matches and 0 if not. Change regex and testdata as appropriate

Re: testing new regex

sherm77 — Wed, 04 Feb 2015 21:42:27 GMT

Splunk has included some very nice field extraction tools, but sometimes they don't work the way it's supposed to, you see it immediately when you send it to a tabular output (ie, table, stats, etc). I just found the pcregextest command, so I'll have to try it out.

David Maislin from Splunk clued me into regex101.com recently and I've used it heavily. It's the best regex tool that I've found. Try it out and let me know if you've found anything better.