Splunk lightforwarder to Index/Search server using custom sourcetype settings

cvImplex — Wed, 02 Mar 2011 07:18:45 GMT

My lightforwarders are working and sending event information to my index/search server but the customer sourcetypes I have created are not parsing the log information. All lightforwarders (2 separate lightforwarders) are standard out-of-the-box configuration with a input data file/directory monitor added and manually set to the sourcetype below. The index/search server is a free trial license out-of-the-box full install with a receiving port setup for the forwarders. I have performed the following setups:

1) Both the lightforwarders and the index server have the changes to the props.conf and transforms.conf files. The data is marked with the correct sourcetype name when I search for it, but the fields are not parsed.

2) the lightforwarders don't have any changes to the props.conf or transforms.conf files. The index server has the changes to the props.conf and transform.conf files. I have the same result as #1.

I have searched thru the documentation and the answers forum and can't seem to find a simple answer other than this http://answers.splunk.com/questions/906/where-is-the-best-place-for-props-conf-and-transforms-conf, which is how I setup the servers in #2.

Any help would be appreciated.

Here are the changes I made to props.conf:

[iis_wms]
pulldown_type = true 
MAX_TIMESTAMP_LOOKAHEAD = 32
SHOULD_LINEMERGE = False
CHECK_FOR_HEADER = True
REPORT-wmsfields = iis-wms-fields
TIME_FORMAT = %Y-%m-%d %H:%M:%S

[iis_wms_short]
pulldown_type = true 
MAX_TIMESTAMP_LOOKAHEAD = 32
SHOULD_LINEMERGE = False
CHECK_FOR_HEADER = True
REPORT-wmsfields = iis-wms-fields-short
TIME_FORMAT = %Y-%m-%d %H:%M:%S

[wowza]
pulldown_type = true 
MAX_TIMESTAMP_LOOKAHEAD = 32
CHECK_FOR_HEADER = True
REPORT-wmsfields = wowza-fields
TIME_FORMAT = %Y-%m-%d %H:%M:%S

Here are the changes to the transforms.conf:

[iis-wms-fields]
DELIMS = " "
FIELDS = "c-ip","date","time","c-dns","cs-uri-stem","c-starttime","x-duration","c-rate","c-status","c-playerid","c-playerversion","c-playerlanguage","cs(User-Agent)","cs(Referer)","c-hostexe","c-hostexever","c-os","c-osversion","c-cpu","filelength","filesize","avgbandwidth","protocol","transport","audiocodec","videocodec","channelURL","sc-bytes","c-bytes","s-pkts-sent","c-pkts-received","c-pkts-lost-client","c-pkts-lost-net","c-pkts-lost-cont-net","c-resendreqs","c-pkts-recovered-ECC","c-pkts-recovered-resent","c-buffercount","c-totalbuffertime","c-quality","s-ip","s-dns","s-totalclients","s-cpu-util","cs-user-name","s-session-id","s-content-path","cs-url","cs-media-name","c-max-bandwidth","cs-media-role","s-proxied"

[iis-wms-fields-short]
DELIMS = " "
FIELDS = "c-ip","date","time","c-dns","cs-uri-stem","c-starttime","x-duration","c-rate","c-status","c-playerid","c-playerversion","c-playerlanguage","cs(User-Agent)","cs(Referer)","c-hostexe","c-hostexever","c-os","c-osversion","c-cpu","filelength","filesize","avgbandwidth","protocol","transport","audiocodec","videocodec","channelURL","sc-bytes","c-bytes","s-pkts-sent","c-pkts-received","c-pkts-lost-client","c-pkts-lost-net","c-pkts-lost-cont-net","c-resendreqs","c-pkts-recovered-ECC","c-pkts-recovered-resent","c-buffercount","c-totalbuffertime","c-quality","s-ip","s-dns","s-totalclients","s-cpu-util"

[wowza-fields]
DELIMS = " "
FIELDS = "date","time","tz","x-event","x-category","x-severity","x-status","x-ctx","x-comment","x-vhost","x-app","x-appinst","x-duration","s-ip","s-port","s-uri","c-ip","c-proto","c-referrer","c-user-agent","c-client-id","cs-bytes","sc-bytes","x-stream-id","x-spos","cs-stream-bytes","sc-stream-bytes","x-sname","x-sname-query","x-file-name","x-file-ext","x-file-size","x-file-length","x-suri","x-suri-stem","x-suri-query","cs-uri-stem","cs-uri-query"

Example of log file I am parsing:

#Software: Windows Media Services
#Version: 4.1
#Date: 2011-03-01 06:00:01
#Fields: c-ip date time c-dns cs-uri-stem c-starttime x-duration c-rate c-status c-playerid c-playerversion c-playerlanguage cs(User-Agent) cs(Referer) c-hostexe c-hostexever c-os c-osversion c-cpu filelength filesize avgbandwidth protocol transport audiocodec videocodec channelURL sc-bytes c-bytes s-pkts-sent c-pkts-received c-pkts-lost-client c-pkts-lost-net c-pkts-lost-cont-net c-resendreqs c-pkts-recovered-ECC c-pkts-recovered-resent c-buffercount c-totalbuffertime c-quality s-ip s-dns s-totalclients s-cpu-util
100.100.100.10 2011-03-01 06:00:43 - - - 0 1 400 - - - - - - - - - - - - - http TCP - - - 0 - 0 - - - - - - - - - - 100.100.100.10  computerhostname 1 14 
100.100.100.10 2011-03-01 06:01:44 - - - 0 1 400 - - - - - - - - - - - - - http TCP - - - 0 - 0 - - - - - - - - - - 100.100.100.10  computerhostname 1 15 
71.177.164.232 2011-03-01 06:13:46 - http://computerhostname/directory/video.wmv 0 14 1 200 {3300AD50-2C39-46c0-AE0A-8D637E18BC07} 11.0.5721.5275 en-US WMFSDK/11.0.5721.5275_WMPlayer/11.0.5721.5268 - wmplayer.exe 11.0.5721.5145 Windows_XP 5.1.0.2600 Pentium 1437 38642962 213517 http TCP Windows_Media_Audio_9 Windows_Media_Video_V7 - 486022 483120 335 333 0 0 0 0 0 0 1 5 100 100.100.100.10  computerhostname 1 14 
71.177.164.232 2011-03-01 06:15:51 - http://computerhostname/directory/video.wmv 1304 124 1 200 {3300AD50-2C39-46c0-AE0A-8D637E18BC07} 11.0.5721.5275 en-US WMFSDK/11.0.5721.5275_WMPlayer/11.0.5721.5268 - wmplayer.exe 11.0.5721.5145 Windows_XP 5.1.0.2600 Pentium 1437 38642962 216687 http TCP Windows_Media_Audio_9 Windows_Media_Video_V7 - 3362650 3362650 2318 2318 0 0 0 0 0 0 1 5 100 100.100.100.10  computerhostname 1 15

My forwarder inputs.conf file:

[default]
host = godzilla

[script://$SPLUNK_HOME\bin\scripts\splunk-regmon.path]
disabled = 1

[script://$SPLUNK_HOME\bin\scripts\splunk-admon.path]
disabled = 1

[script://$SPLUNK_HOME\bin\scripts\splunk-wmi.path]
disabled = 1

/etc/apps/learned/local/props.conf:

[iis_wms_short-2]
KV_MODE = none
MAX_TIMESTAMP_LOOKAHEAD = 32
REPORT-AutoHeader = AutoHeader-1
REPORT-wmsfields = iis-wms-fields-short
SHOULD_LINEMERGE = False
given_type = iis_wms_short
pulldown_type = true

Re: Splunk lightforwarder to Index/Search server using custom sourcetype settings

gkanapathy — Wed, 02 Mar 2011 07:55:45 GMT

What does the inputs.conf stanza on your forwarder look like?

Re: Splunk lightforwarder to Index/Search server using custom sourcetype settings

gkanapathy — Wed, 02 Mar 2011 07:59:38 GMT

Also, do you have anything in $SPLUNK_HOME/etc/apps/learned/local/props.conf? Anything with the names of those sourcetypes, in particular?

Re: Splunk lightforwarder to Index/Search server using custom sourcetype settings

cvImplex — Wed, 02 Mar 2011 23:40:26 GMT

learned/local/props.conf contains the following (which I would love to prevent):

[iis_wms_short-2]
KV_MODE = none
MAX_TIMESTAMP_LOOKAHEAD = 32
REPORT-AutoHeader = AutoHeader-1
REPORT-wmsfields = iis-wms-fields-short
SHOULD_LINEMERGE = False
given_type = iis_wms_short
pulldown_type = true

Re: Splunk lightforwarder to Index/Search server using custom sourcetype settings

cvImplex — Fri, 04 Mar 2011 05:11:57 GMT

Answer to my question:

1) Removed the "CHECK_FOR_HEADER" in props.conf because it was causing duplicate sourcetypes in the learned/local/props.conf file.

2) Each lightforwarder and index/search needs the same props.conf and transforms.conf files in the system/local folder.

3) Installed the windows app into my receiver which is linux.

Re: Splunk lightforwarder to Index/Search server using custom sourcetype settings

bpravisa — Sat, 29 Sep 2012 04:18:52 GMT

I just would like to point out that the use of the "-" (hyphen) on stanza names on the transforms.conf may cause some issues, like ignoring the transform:

[wowza-fields]
DELIMS = " "
FIELDS = "date","time","tz","x-event","x-category","x-severity","x-status","x-ctx","x-comment",...

Rather than that use underscore (Ex: "[wowza_fields]") and save lots of time : )
In my experience SPLUNK ignored the stanza defined on transforms.conf and the fields were not seen on the "field discovery" panel. Hope this helps!

topic Re: Splunk lightforwarder to Index/Search server using custom sourcetype settings in Getting Data In

Splunk lightforwarder to Index/Search server using custom sourcetype settings

Re: Splunk lightforwarder to Index/Search server using custom sourcetype settings

Re: Splunk lightforwarder to Index/Search server using custom sourcetype settings

Re: Splunk lightforwarder to Index/Search server using custom sourcetype settings

Re: Splunk lightforwarder to Index/Search server using custom sourcetype settings

Re: Splunk lightforwarder to Index/Search server using custom sourcetype settings