Dataset
10.24.11.102 - user1 [10/Sep/2016:02:46:12 -0400] "GET http://www.foo.org:80/lib/stone/csrf/token.json HTTP/1.1" 200 393
10.32.52.18 - user2 [10/Sep/2016:02:28:21 -0400] "GET https://aaa.idm.purple.org:8443/login HTTP/1.1" 200 2049
10.210.18.17 - - [10/Sep/2016:00:10:57 -0400] "GET http://explore.google.org/robots.txt HTTP/1.1" 200 2049
10.31.2.124 - user3 [09/Sep/2016:21:04:47 -0400] "POST http://bar.tree.com:80/authn-callback HTTP/1.1" 200 1562
When I search for
index=library sourcetype=proxy_access
I do not get back ** method,url,protocol ** which would come from ** data_from_method_url**
When I search for
index=library sourcetype=proxy_access | extract reload=T
| extract ProzyData
| extract data_from_method_url
method, url, and protocol are all extracted correctly.
The first extraction REPORT-Extract is working as I get all of the expected fields.
GET http://www.foo.org:80/lib/stone/csrf/token.json HTTP/1.1
GET https://aaa.idm.purple.org:8443/login HTTP/1.1
GET http://explore.google.org/robots.txt HTTP/1.1
POST http://bar.tree.com:80/authn-callback HTTP/1.1
How do I get the method, url, and protocol to extract using the props and transforms.
I have done many version of these files, but this is how they currently read.
props.conf
[proxy_access]
REPORT-Extract = ProzyData
description = Access Logs
KV_MODE = none
[pull_from_method_url]
REPORT-method_from_method_url = data_from_method_url
transforms.conf
[ProzyData]
DELIMS = " "
FIELDS = "src_ip","Unknown","user","datetime","timeoffset","method_url","responce","bytes"
################ extract from source_key #############
[data_from_method_url]
SOURCE_KEY = method_url
DELIMS = " "
FIELDS = method,url,protocol
In your props.conf you have a stanza named pull_from_method_url. This settings under here should be under the same stanza at the other transform, proxy_access, as this is the sourcetype of your data. Stanza headings should be either sourcetype, source or host - unless I am misunderstanding and your data does have the sourcetype of pull_from_method_url?