topic Re: transforms.conf in Getting Data In

transforms.conf

shangshin — Tue, 29 May 2012 20:05:27 GMT

[my_fields]
REGEX = ^[[nspaces:clientip]]\s++[[nspaces:ident]]\s++[[nspaces:user_id]]\s++[[sbstring:req_time]]\s++[[qstring:method_url_protocol]]\s++[[nspaces:status]]\s++[[nspaces:bytes]]\s++[[qstring:referer_url]]\s++[[qstring:useragent]]

[method_url_protocol]
DELIMS = " "
FIELDS = method, url, protocol

Hi,
I define these 2 stanzas above in transforms.conf and expect to extract some info from web access log. As you can see the sample quoted string below, it contains 3 fields. However, these 3 fields are not extracted our successsfully. Can you shed some light on it?

"POST /amazon.com/view.do HTTP/1.1"

Re: transforms.conf

Ayn — Tue, 29 May 2012 20:12:46 GMT

Are you referring to these transforms in props.conf?

Re: transforms.conf

shangshin — Mon, 28 Sep 2020 11:52:38 GMT

yes, that's right. In props.conf,

[access_log_reg]
NO_BINARY_CHECK = 1
pulldown_type = 1
REPORT-myfields = my_fields

Re: transforms.conf

Ayn — Mon, 28 Sep 2020 11:52:40 GMT

What about the method_url_protocol transform? That's the one that, if configured properly, would do the work.

Re: transforms.conf

shangshin — Mon, 28 Sep 2020 11:52:43 GMT

here is the complete sample roq in web access log
10.39.208.2 - clinet_user_id [29/May/2012:14:04:10 -0400] "POST /amazon.com/view.do HTTP/1.1" 200 1214 "google.com" "Java/1.5.0_06"

As you can, the field of method, url and protocol can be extracted out as a single value using the first stanza (my_fields). However, the second stanza (method_url_protocol) is unable to parse the value. I guess I didn't set it up properly...

Re: transforms.conf

Ayn — Tue, 29 May 2012 20:56:42 GMT

This is why I'm asking if you're actually referring to that transform from props.conf. If you just setup the transform but don't refer to it anywhere, it won't ever be applied.

Re: transforms.conf

richprescott — Mon, 28 Sep 2020 11:55:28 GMT

I think Ayn is referring to adding the regex as follows in props.conf.

[access_log_reg] NO_BINARY_CHECK = 1 pulldown_type = 1 REPORT-myfields = my_fields,method_url_protocol

Re: transforms.conf

jpmasseria — Mon, 28 Sep 2020 15:00:26 GMT

Try using [[access-request]] to extract method,uri and version.

Below is what I used and it worked for me.

REGEX =
^[[nspaces:clientip]]\s++[[nspaces:ident]]\s++[[nspaces:user_id]]\s++[[sbstring:req_time]]\s++[[access-request]]\s++[[nspaces:status]]\s++[[nspaces:bytes]]\s++[[qstring:referer_url]]\s++[[qstring:useragent]]\s++[[qstring:someurl]]\s++[[nspaces:response_time]]