transforms with SOURCE_KEY using FIELDS

willamwar — Tue, 29 Sep 2020 11:28:11 GMT

Dataset

10.24.11.102 - user1 [10/Sep/2016:02:46:12 -0400] "GET http://www.foo.org:80/lib/stone/csrf/token.json HTTP/1.1" 200 393
10.32.52.18 - user2 [10/Sep/2016:02:28:21 -0400] "GET https://aaa.idm.purple.org:8443/login HTTP/1.1" 200 2049
10.210.18.17 - - [10/Sep/2016:00:10:57 -0400] "GET http://explore.google.org/robots.txt HTTP/1.1" 200 2049
10.31.2.124 - user3 [09/Sep/2016:21:04:47 -0400] "POST http://bar.tree.com:80/authn-callback HTTP/1.1" 200 1562

When I search for
index=library sourcetype=proxy_access

I do not get back ** method,url,protocol ** which would come from ** data_from_method_url**

When I search for

index=library sourcetype=proxy_access | extract reload=T
| extract ProzyData
| extract data_from_method_url

method, url, and protocol are all extracted correctly.

The first extraction REPORT-Extract is working as I get all of the expected fields.
GET http://www.foo.org:80/lib/stone/csrf/token.json HTTP/1.1
GET https://aaa.idm.purple.org:8443/login HTTP/1.1
GET http://explore.google.org/robots.txt HTTP/1.1
POST http://bar.tree.com:80/authn-callback HTTP/1.1

How do I get the method, url, and protocol to extract using the props and transforms.

I have done many version of these files, but this is how they currently read.

props.conf
  [proxy_access]
  REPORT-Extract = ProzyData
  description = Access Logs
  KV_MODE = none

  [pull_from_method_url]
  REPORT-method_from_method_url = data_from_method_url

transforms.conf
  [ProzyData]
  DELIMS = " "
  FIELDS =    "src_ip","Unknown","user","datetime","timeoffset","method_url","responce","bytes"

  ################ extract from source_key #############
  [data_from_method_url]
  SOURCE_KEY = method_url
  DELIMS = " "
  FIELDS = method,url,protocol

Re: transforms with SOURCE_KEY using FIELDS

lquinn — Tue, 29 Sep 2020 11:33:15 GMT

In your props.conf you have a stanza named pull_from_method_url. This settings under here should be under the same stanza at the other transform, proxy_access, as this is the sourcetype of your data. Stanza headings should be either sourcetype, source or host - unless I am misunderstanding and your data does have the sourcetype of pull_from_method_url?

topic transforms with SOURCE_KEY using FIELDS in Splunk Search

transforms with SOURCE_KEY using FIELDS

Re: transforms with SOURCE_KEY using FIELDS