topic Re: W3C Fields With Light Forwarder - Still don't have it in Getting Data In

W3C Fields With Light Forwarder - Still don't have it

dveith — Fri, 24 Sep 2010 07:18:21 GMT

Please advise.

Linux Splunk Server 4.1.5 Light forwarder is installed on Windows IIS web Servers Trying to get W3C Extended fields available for searching on the Splunk Server. the data is forwarded, just not with fields defined.

Windows IIS Servers have this inputs.conf

[default]
host = servername

[monitor://C:\WINNT\system32\LogFiles\W*\ex*.log]
SOURCETYPE = iis

Records also display with source types "IIS" "IIS-1" IIS-5" on the Splunk server.

What the best way to configure this to the IIS logs have their W#C Extended fields available for searching?

thanks.

Re: W3C Fields With Light Forwarder - Still don't have it

gkanapathy — Fri, 24 Sep 2010 10:15:59 GMT

Yeah. So, the default settings in props.conf don't work well if you're using a forwarder for IIS log files, which admittedly a lot of people do (and should do).

Here's what I would do. First, SOURCETYPE should be sourcetype, i.e., lower-case. Next, on the forwarder (where the input phase occurs, reference) add this to a props.conf next to your inputs.conf:

 [iis]
 CHECK_FOR_HEADER = false

Then, on the search head, configure your fields manually in a props.conf:

 [iis]
 REPORT-iisfields = iisfields

and transforms.conf

[iisfields]
DELIMS = " "
FIELDS = date,time,csWhatever,csWhatever2,csNextField,scMoreInfo

If you have multiple different sets of fields (e.g., different servers/instances/sites log different fields), then specify a different sourcetype for them in inputs, and define different fields for it on the search head.

Re: W3C Fields With Light Forwarder - Still don't have it

araitz — Fri, 24 Sep 2010 10:56:24 GMT

Why is this still painful? Amazing....

Re: W3C Fields With Light Forwarder - Still don't have it

dveith — Fri, 24 Sep 2010 22:55:34 GMT

Thank you for your excellent response. We do have different sets of fields for different web sites on the same IIS servers so we will need to specify multiple sourcetypes and fields. Thanks for that tip too.

It's things like this that still make me feel that Windows is still a second-class citizen to Splunk.

Re: W3C Fields With Light Forwarder - Still don't have it

dveith — Sat, 25 Sep 2010 04:24:37 GMT

gkanapathy, this worked well. Except the header records make it through now too. How can I eliminate them?

Re: W3C Fields With Light Forwarder - Still don't have it

gkanapathy — Sat, 25 Sep 2010 04:45:02 GMT

You can mostly just ignore them, or construct your search queries to ignored them (e.g., NOT user=csUser or whatever will exclude those items) Otherwise you can do a TRANSFORMS at index time and filter out (nullQueue) anything that matches ^#

Re: W3C Fields With Light Forwarder - Still don't have it

gkanapathy — Sat, 25 Sep 2010 04:46:31 GMT

I would also encourage you to file an enhancement request (aka P4 support ticket) on this. This is something that Splunk should fix, and if the "Getting Data In" tasks for the next version do anything at all, it should deal with this issue.

Re: W3C Fields With Light Forwarder - Still don't have it

dveith — Sat, 25 Sep 2010 05:13:36 GMT

Hi, I will submit an enhancement request. And before I saw you note I got it working sending to a null queue. Thanks for your help!!