Field extraction of custom entry

[jdziedzic]

I have a unique entry (custom.gif) in an indexed sourcetype that need's to extract fields via a "&" delimiter while also using the automatic field extraction. Is this possible?

Example:
sourcetype=my_log "GET /custom.gif"

172.16.0.1 - - [28/Apr/2014:12:00:51 -0500] "GET /custom.gif?type=timings&url=httttttp%3A%2F%2Fwww.barf.com%2F&loadEvent=19153&domContentLoadedEvent=5002&domInteractive=4996&domainLookup=0&connect=0&response=1325&request=6&fetchStart=5 httttttp/1.0" 200 35 "?url=httttttp%3A%2F%2Fwww.barf.com%2F&loadEvent=19153&domContentLoadedEvent=5002&domInteractive=4996&domainLookup=0&connect=0&response=1325&request=6&fetchStart=5" "httttttp://www.barf.com/" "Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; rv:11.0) like Gecko" "D_SID=69.35.163.244; D_PID=456AB-AE82-3F22-B757-C761BE1C4; D_UID=1C4-3F85-302C-A2C1-145315D1D4; D_IID=12C8-F325-3DCB-8954-47D75A73340E; timed-modal=139846694; __utma=648177734019904.1398704444.1398704444.104444.1; __utmb=64817776.2.9.13945828; __utmc=64817776; __utmz=64817798704444.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); s_cc=true; gpv_pn=cc%3Ahome%3Ahomepage; s_sq=%5B%5BB%5D%5D"  192.168.0.1 - 0.001 somehost "-"

These are the fields that splunk was able to create from this entry. Got this by adding this at the end of the search:
| stats dc(*) as *

D_IID

D_PID

D_SID

D_UID

auth_user

cookie

date

date_hour

date_mday

date_minute

date_month

date_second

date_wday

date_year

date_zone

domain

gpv_pn

has_qs

host

httpvers

index

linecount

method

punct

qs

referrer

s_cc

s_sq

size

source

source_ip

sourcetype

splunk_server

srcip

srcip2

status

timed_modal

timeendpos

timestartpos

ua

uri

utma

utmb

utmc

So basically I want splunk to do its auto field extraction thing in addition to extracting "&" delimited fields but only for the custom.gif entries.

We are using version 5.03

[somesoni2]

Try this for your data input

props.conf

[my_log]
pulldown_type = true 
maxDist = 28
MAX_TIMESTAMP_LOOKAHEAD = 128
SHOULD_LINEMERGE = False
TIME_PREFIX = \[
REPORT-yourExtract = youlog-extracts

transforms.conf

[youlog-extracts]
REGEX = ^[[nspaces:clientip]]\s++[[nspaces:ident]]\s++[[nspaces:user]]\s++[[sbstring:req_time]]\s++[[access-request]]\s++[[nspaces:status]]\s++[[nspaces:bytes]](?:\s++"(?<referer>[[bc_domain:referer_]]?+[^"]*+)"(?:\s++[[qstring:useragent]](?:\s++[[qstring:cookie]])?+)?+)?[[all:other]]

[jdziedzic]

This does not work. I've tried multiple .conf's and attempted to test it via a search.

To simplify my request:

How do I parse this with "&" start and "=" end separator and have Splunk auto define the field's between the 2 parse's, making the value of these searchable?

If your answer is the same, perhaps if you could break down the REGEX, I might have a better understanding on how this works.

Thanks!

[somesoni2]

You can also use REST API endpoint to get the same. e.g. query

|rest /services/data/props/extractions | table title, stanza, type , eai:acl.app, value
|join value type=left [|rest /services/data/transforms/extractions | table title eai:appName | rename title as value] | rename eaiappName as TransformAppName, stanza as sourcetype, eai:acl.app as PropsAppName

If you change the content of props.conf or transforms.conf, you need either splunk restart or splunk refresh (launch URL http://yourhost:8000/en-US/debug/refresh)

[somesoni2]

You can see which fields extractions or transforms are present under which app from Splunk Web UI (Manager-> Fields -> Field extractions OR Field Transformations. Look for column App and the props.conf/transforms.conf should inside that app folder (if App=system, its in etc/system).

[somesoni2]

This setting will extract fields, regardless of uri (custom.gif or any other value). I am not sure if its possible to have field extraction based on a filter. To my knowledge its a sourcetype specific configuration.

This should work for existing entry as well.

[somesoni2]

Can you provide your current props/transforms conf file values?

[somesoni2]

The configuration that I have provided is generic enough to handle addition of "&" separated fields (tested).

[somesoni2]

Are you not getting "custom.gif" value in the uri field?