topic Re: searchtime field extraction, ignore event prefix in Splunk Search

searchtime field extraction, ignore event prefix

atat23 — Mon, 02 Dec 2013 16:34:20 GMT

Hi all,

I'm using props and transforms to extract fields, all the fields are extracted properly, except the first one.

This is mainly because my transforms is currently:

[extract_fields]
DELIMS = ","
FIELDS = "field1", "field2", "field3", etc

The first line of the event itself is as follows:

Dec 2 15:47:55 foo.bar.ie IRL:"field1","field2","field3"

So it extracts Dec 2 15:47:55 foo.bar.ie IRL:"field1" as the first field.

How would I go about ignoring that first part, so the field extraction starts after IRL:?

I was looking for some kind of field extraction parameter similar to TIME_PREFIX but for field extractions, but there doesn't seem to be one or I'm just not understanding the docs correctly.

Re: searchtime field extraction, ignore event prefix

somesoni2 — Mon, 02 Dec 2013 18:17:03 GMT

Alternative using just the props.conf.

[extract_fields]
EXTRACT-fieldval = IRL:(?<field1>.*),(?<field2>.*),(?<field3>.*)

more fields can be added by appending ",(?.*)" in the end. See if this solution is feasible for you.

Re: searchtime field extraction, ignore event prefix

atat23 — Mon, 02 Dec 2013 19:09:01 GMT

I tired this, both with the field surrounded by quotes and without as that's the way they are in the log, although using this method it doesn't seem to be extracting any fields.

edit: should also mention, i also tried to name the stanza as [source_log] in props, which is my sourcetype I want this applied to.

Re: searchtime field extraction, ignore event prefix

somesoni2 — Tue, 03 Dec 2013 00:22:28 GMT

I hope you have restarted your splunk instance after changing the props.conf file. For troubleshooting, you can try running the provided regex in search using "|rex" command. "index=yourindex sourcetyo=pe=source_log | rex \"paste the regex"

Re: searchtime field extraction, ignore event prefix

lguinn2 — Tue, 03 Dec 2013 07:58:16 GMT

You do not need to restart splunk if your props.conf changes are only field extractions. But if you want to explicitly reload the field extractions, you can run this command

| extract reload=true

Yes, this search string starts with a |

Re: searchtime field extraction, ignore event prefix

lguinn2 — Tue, 03 Dec 2013 08:05:18 GMT

Maybe the regex needs to be revised to this

EXTRACT-fieldval = IRL:(?<field1>.*?),(?<field2>.*?),(?<field3>.*?)

as the first regex is "greedy" and that sometimes causes problems. Also, this regex assumes that there are no actual quotation marks in the data.

A final thought: there are rules for field names. A field name can contain only letters, numbers and the underscore character. It must begin with a letter. If your names were violating these rules, that could be a problem. Yes, field names can have special characters in some places if you use quotation marks - but I wouldn't do it.

Re: searchtime field extraction, ignore event prefix

atat23 — Tue, 03 Dec 2013 16:19:40 GMT

I was restarting after every change. Using rex the fields seem to extract fine, not in props though, so:

rex "ASM:.(?<field1>[^,]*).,.(?<field2>[^,]*).

props
EXTRACT-fieldval = ASM:.(?<field1>[^,]*).,.(?<field2>[^,]*).

so maybe the problem lies with the fact my fields do have literal quotes around them, and I'm trying to match everything between the quotes.

Above I match the quotation marks with . instead of matching the literal quotation marks. I also tried the two suggested regexes, work with rex, not props.

permissions seem to be fine on the files as well.

Re: searchtime field extraction, ignore event prefix

atat23 — Tue, 03 Dec 2013 18:33:48 GMT

so length seems to be an issue (hur hur), I'm trying to extract 35 fields, so the line in props is pretty long.

When I test it using only the first five fields, it works and extracts these fields as it should.

Is there a limit to the number of fields to be extracted in props....? am I reaching a character limit meaning the whole line is ignored?