testing new regex

[njathan]

Before really putting my custom regex in transforms.conf, is there a quick way to test and debug it?

[sherm77]

Splunk has included some very nice field extraction tools, but sometimes they don't work the way it's supposed to, you see it immediately when you send it to a tabular output (ie, table, stats, etc). I just found the pcregextest command, so I'll have to try it out.

David Maislin from Splunk clued me into regex101.com recently and I've used it heavily. It's the best regex tool that I've found. Try it out and let me know if you've found anything better.

[tpflicke]

Here's a quick way to test a regex in splunk by supplying test data via an eval, e.g. running a query like:

index=* | head 1 | eval testdata="123456a" | regex testdata="^.{5,6}(?<!p)a$" | stats count

The result is 1 if the regex matches and 0 if not. Change regex and testdata as appropriate

[gkanapathy]

I usually use $SPLUNK_HOME/bin/pcregextest on the Splunk command line because it uses the exact regex engine as comes with Splunk. The command line is somewhat inconvenient and you do have to make sure that you quote a few characters correctly according to shell quoting rules. (Please note that you need to quote characters when using rex as well, and rex quoting rules are an even worse headache. Furthermore, bash shell quoting is well-defined and completely documented.)

[Lowell]

Good point about rex and the quoting considerations. Generally I've only hit issues when I have to match a literal " (double-quote) character, but it's important to be aware that you can't always just copy and paste between a rex command and a config file.

[southeringtonp]

You may also want to look at Kodos (free) or RegexBuddy (commercial).

[Lowell]

I normally end up using Kodos for testing. The only down side is that it uses Python regular expression syntax instead of pcre-style regular expressions. The only difference I've see are: (1) You have to add a P in a capture group name, like: (?P<group>..) Of course, PCRE accepts both, so I often let the extra "P" in the final regex. And (2), and python (and therefore Kodos) doesn't support atomic grouping or possessive quantifiers (like \s++). But this generally hasn't been too much of a problem since I don't often use these advanced regex these features.

[Ledion_Bitincka]

Use the rex command: ..... | rex "" | .....,

For example use the following regex to extract indexes.conf into the conf_file variable in the following event:

05-21-2010 17:41:51.166 INFO  IndexProcessor - indexes.conf - memPoolMB param autotuned to 512MB
... | rex " - (?<conf_file>\w+\.conf) - "| .....

[Ledion_Bitincka]

@njathan - I was talking about testing the regex using the rex search command - either from splunk's search UI or CLI

[rroberts]

Searching the internal index for regex extraction processor errors may help too. Like so...

index=_internal regexExtractionProcessor ERROR

7:59:13.931 AM

07-30-2010 07:59:13.931 ERROR regexExtractionProcessor - Cannot compile RE "((.AcctCode=\d{5}-)." for trade: missing )
host=class4.splunk.com Options| sourcetype=splunkd Options| source=/home/rick4uc/opt/splunk/var/log/splunk/splunkd.log Options

[njathan]

i assume you are talking about using the splunk command line, are you? how do i invoke the splunk command line? (sorry for sounding so n00by! installed splunk only yesterday 😛 )