I have a unique entry (custom.gif) in an indexed sourcetype that need's to extract fields via a "&" delimiter while also using the automatic field extraction. Is this possible?
Example:
sourcetype=my_log "GET /custom.gif"
172.16.0.1 - - [28/Apr/2014:12:00:51 -0500] "GET /custom.gif?type=timings&url=httttttp%3A%2F%2Fwww.barf.com%2F&loadEvent=19153&domContentLoadedEvent=5002&domInteractive=4996&domainLookup=0&connect=0&response=1325&request=6&fetchStart=5 httttttp/1.0" 200 35 "?url=httttttp%3A%2F%2Fwww.barf.com%2F&loadEvent=19153&domContentLoadedEvent=5002&domInteractive=4996&domainLookup=0&connect=0&response=1325&request=6&fetchStart=5" "httttttp://www.barf.com/" "Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; rv:11.0) like Gecko" "D_SID=69.35.163.244; D_PID=456AB-AE82-3F22-B757-C761BE1C4; D_UID=1C4-3F85-302C-A2C1-145315D1D4; D_IID=12C8-F325-3DCB-8954-47D75A73340E; timed-modal=139846694; __utma=648177734019904.1398704444.1398704444.104444.1; __utmb=64817776.2.9.13945828; __utmc=64817776; __utmz=64817798704444.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); s_cc=true; gpv_pn=cc%3Ahome%3Ahomepage; s_sq=%5B%5BB%5D%5D" 192.168.0.1 - 0.001 somehost "-"
D_IID
D_PID
D_SID
D_UID
auth_user
cookie
date
date_hour
date_mday
date_minute
date_month
date_second
date_wday
date_year
date_zone
domain
gpv_pn
has_qs
host
httpvers
index
linecount
method
punct
qs
referrer
s_cc
s_sq
size
source
source_ip
sourcetype
splunk_server
srcip
srcip2
status
timed_modal
timeendpos
timestartpos
ua
uri
utma
utmb
utmc
We are using version 5.03
Try this for your data input
props.conf
[my_log]
pulldown_type = true
maxDist = 28
MAX_TIMESTAMP_LOOKAHEAD = 128
SHOULD_LINEMERGE = False
TIME_PREFIX = \[
REPORT-yourExtract = youlog-extracts
transforms.conf
[youlog-extracts]
REGEX = ^[[nspaces:clientip]]\s++[[nspaces:ident]]\s++[[nspaces:user]]\s++[[sbstring:req_time]]\s++[[access-request]]\s++[[nspaces:status]]\s++[[nspaces:bytes]](?:\s++"(?<referer>[[bc_domain:referer_]]?+[^"]*+)"(?:\s++[[qstring:useragent]](?:\s++[[qstring:cookie]])?+)?+)?[[all:other]]
This does not work. I've tried multiple .conf's and attempted to test it via a search.
To simplify my request:
How do I parse this with "&" start and "=" end separator and have Splunk auto define the field's between the 2 parse's, making the value of these searchable?
If your answer is the same, perhaps if you could break down the REGEX, I might have a better understanding on how this works.
Thanks!
You can also use REST API endpoint to get the same. e.g. query
|rest /services/data/props/extractions | table title, stanza, type , eai:acl.app, value
|join value type=left [|rest /services/data/transforms/extractions | table title eai:appName | rename title as value] | rename eaiappName as TransformAppName, stanza as sourcetype, eai:acl.app as PropsAppName
If you change the content of props.conf or transforms.conf, you need either splunk restart or splunk refresh (launch URL http://yourhost:8000/en-US/debug/refresh)
You can see which fields extractions or transforms are present under which app from Splunk Web UI (Manager-> Fields -> Field extractions OR Field Transformations. Look for column App and the props.conf/transforms.conf should inside that app folder (if App=system, its in etc/system).
This setting will extract fields, regardless of uri (custom.gif or any other value). I am not sure if its possible to have field extraction based on a filter. To my knowledge its a sourcetype specific configuration.
This should work for existing entry as well.
Can you provide your current props/transforms conf file values?
The configuration that I have provided is generic enough to handle addition of "&" separated fields (tested).
Are you not getting "custom.gif" value in the uri field?