topic Having REGEX Problems in Splunk Search

Having REGEX Problems

usersnation — Tue, 21 Sep 2010 06:04:54 GMT

Sorry for the cross post but after posting i saw a recommendation to use this forum instead of splunk.com

I am having issues getting REGEX to work my sample input is a simple two column csv for testing purposes (date,fakeip) the data looks like this:

8/31/2010 12:12:12,1.1.1.1
9/1/2010 12:12:12,1.1.1.1
9/2/2010 12:12:12,1.1.1.1
9/3/2010 12:12:12,1.1.1.1

my config files are as follows

props.conf

[c_netflowdata]
TRANSFORMS-foo=c_regex

Over the course of the past 8 hours I have tried a large number of different transforms files none which seem to do anything useful so there is obviously something that i am missing from the splunk documentation and peoples examples on this forum below are a few transforms.conf examples that i have tried. The have all produce varying results none of which have resulted in selectable fields in the Search app's field pick'em box

transforms.conf

[c_regex]
REGEX=(\d+/\d+/\d+\s\d+:\d+:\d+)
FORMAT=thedate::$1

[c_regex]
REGEX=(\d+/\d+/\d+\s\d+:\d+:\d+).*
FORMAT=thedate::$1

[c_regex]
REGEX=[^,],(\d+\.\d+\.\d+\.\d+)    
FORMAT=src_ip::$1

[c_regex]
REGEX=(\d+/\d+/\d+\s\d+:\d+:\d+),(\d+\.\d+\.\d+\.\d+)
FORMAT=thedate::$1 src_ip::$2

I have successfully parsed the data using DELIMS and FIELDS - however there is a specific reason that I am trying to use REGEX - i.e. this is the first step in solving a larger issue that i am trying to address

Any help is appreciated, thanks in advance

Re: Having REGEX Problems

Stephen_Sorkin — Tue, 21 Sep 2010 06:25:17 GMT

When you specify your regex using TRANSFORMS- in props.conf, this is run at index time and has different properties than search time regexes (specifically, the output needs to be written to _meta with WRITE_META=true). You can either specify it as REPORT- in props.conf, and refer to transforms.conf. It's even easier to specify inline in props.conf:

[c_netflowdata]
EXTRACT-foo = (?<thedate>\d+/\d+/\d+\s\d+:\d+:\d+),(?<src_ip>\d+\.\d+\.\d+\.\d+)

Re: Having REGEX Problems

Genti — Tue, 21 Sep 2010 06:33:38 GMT

Yeap, this should work: [c_regex]
REGEX=(\d+/\d+/\d+\s\d+:\d+:\d+)
FORMAT=thedate::$1
WRITE_META=true

Make sure that the data is actually coming in with sourcetype c_netflow.

Re: Having REGEX Problems

gkanapathy — Tue, 21 Sep 2010 07:19:26 GMT

Do you really want thedate as a field, or is that just an example? If your data has been read in correctly, you should be able to use _time and format it out in any way you want, without using that. Otherwise, see answers below.

Re: Having REGEX Problems

usersnation — Tue, 21 Sep 2010 20:13:23 GMT

yes thedate is just an example sorry for the confusion

Re: Having REGEX Problems

usersnation — Tue, 21 Sep 2010 20:20:39 GMT

I haven't attempted REPORT - but EXTRACT gets the job done. Thanks!

Re: Having REGEX Problems

Genti — Wed, 22 Sep 2010 04:51:52 GMT

then you definitely do not want to use index time extractions