How to index W3C IISlog from a Universal forwarder

[meskildsen]

I am new to Splunk, so please forgive me if the answer to the question is obvious....

I am trying to index W3C IISlogs with splunk. First I tried to index the file locally on the Splunk server. I found a post that suggested that I should add the lines below to

[iisw3c]

pulldown_type = true

MAX_TIMESTAMP_LOOKAHEAD = 32

SHOULD_LINEMERGE = False

CHECK_FOR_HEADER = False

TZ = GMT

REPORT-iisw3cfields = iisw3cfields

TRANSFORMS-removecomments = removecomments

I also added these lines to transforms.conf

[removecomments]

REGEX = ^#.*

DEST_KEY = queue

FORMAT = nullQueue

[iisw3cfields]

DELIMS = " "

FIELDS = date,time,cs-method,cs-uri-stem,cs-uri-query,c-ip,sc-status,sc-substatus,sc-win32-status,sc-bytes,cs-bytes,time-taken

After that it works perfectly as long as I monitor a fodler on the Splunk server and uses my newly defined sourcetype.

Now I want the monitor to run on another remote server and then let the splunk server listen on a TCP port.
I can easily get something back, but now it doesn't recognise the format again. Even if I tell the listener that it is iis3wc.

I found a suggestion that I should add the Checkforheader = fasle on the forwarder, but I am uncertain where exactly to do it since the file structure is slightly different.

Any suggestions of what I have done wrong or what is missing?

[GArienti]

on the UF inputs.conf , how does this differe from

alwaysOpenFile = true

??
is it the same as "FollowTail" ?

[GArienti]

I looked at the splunk page and answered my own question, thank you anyway

followTail = [0|1]
* WARNING: Use of followTail should be considered an advanced administrative
action.
* Treat this setting as an 'action':
* Enable this setting and start the Splunk software.
* Wait enough time for the input to identify the related files.
* Disable the setting and restart.
* DO NOT leave followTail enabled in an ongoing fashion.
* Do not use followTail for rolling log files (log files that get renamed as
they age), or files whose names or paths vary.
* You can use this to force the input to skip past all current data for a
given stanza.
* In more detail: this is intended to mean that if you start the monitor
with a stanza configured this way, all data in the file at the time it is
first encountered will not be read. Only data that arrives after the first
encounter time will be read.
* This can be used to "skip over" data from old log files, or old portions of
log files, to get started on current data right away.
* If set to 1, monitoring starts at the end of the file (like tail -f).
* If set to 0, monitoring starts at the beginning of the file.
* Defaults to 0.

alwaysOpenFile = [0|1]
* Opens a file to check whether it has already been indexed, by skipping the
modification time/size checks.
* Only useful for files that do not update modification time or size.
* Only known to be needed when monitoring files on Windows, mostly for
Internet Information Server logs.
* This flag should only be used as a last resort, as it increases load and
slows down indexing.
* Defaults to 0.

[wrangler2x]

On our instance we are taking logs from the fields given above don't exactly match what we have. Also, we found that the system puts a header at the top of each new daily log file and the header names the fields. Using this, we have this FIELDS statement in our transforms.conf:

FIELDS = date,time,s-ip,cs-method,cs-uri-stem,cs-uri-query,s-port,cs-username,c-ip,c-useragent, sc-status,sc-substatus,sc-win32-status,time-taken

This is working fine.

[meskildsen]

I have tried to reinstall and both Splunk Indexer and Universal Forwarder with no success.
Then I have tried to install a Splunk Indexer instance instead of the Universal Forwarder and configured it as a Heavy Forwarder. This setup was successfull parsed at the Splunk Indexer.
Since I am doing a POC on Splunk the Heavy Forwarder is OK for now.

[okrabbe_splunk]

what does your inputs.conf look like on the forwarder?

[bmacias84 · Accepted solution]

@meskildsen, All you need to do is modify your inputs.conf on the UF similar to this.

UF inputs.conf

[monitor://D:\Logs\Web\...\*.log]

ignoreOlderThan = 3d

disabled = 0

followTail = 0

index = <YourIndex>

sourcetype = iisw3c

UF outputs.conf

[tcpout]

defaultGroup = default-group

[tcpout:default-group]

disabled = 0

server =  <YourIndexer>:9999

indexer inputs.conf

[tcp://:9997]

indexer props.conf

[iisw3c]

pulldown_type = true

MAX_TIMESTAMP_LOOKAHEAD = 32

SHOULD_LINEMERGE = False

CHECK_FOR_HEADER = False

TZ = GMT

REPORT-iisw3cfields = iisw3cfields

TRANSFORMS-removecomments = removecomments

Indexer transforms.conf

[removecomments]

REGEX = ^#.*

DEST_KEY = queue

FORMAT = nullQueue

[iisw3cfields]

DELIMS = " "

FIELDS = date,time,cs-method,cs-uri-stem,cs-uri-query,c-ip,sc-status,sc-substatus,sc-win32-status,sc-bytes,cs-bytes,time-taken

The indexer will pickup the iisw3c sourcetype and apply the settings defined in your props.conf file on your indexer. You will need to configure your outputs.conf to forward the data to your indexer if you haven't already.

Hope this helps.

Also I would recommend reading Howindexingworks and GettingDataIn this will give you a greater understanding on where you should be modifing your .conf files.

[wrangler2x]

I found that the transforms for removing comments did not work on the indexer. I moved them to the forwarder and then it worked.

[bmacias84]

@meskildsen, Pending you dont have any firewall issues the update to my answer will work. Also read http://docs.splunk.com/Documentation/Splunk/latest/admin/Wheretofindtheconfigurationfiles you should learn about conf file precedence.

[meskildsen]

I found the answer to my third question.
outputs.conf is specific to forwarders.

I have defined a tcp forwarder in outputs.conf:

[tcpout-server://SERVERNAME:9999]

This will forward data to the server SERVERNAME and use port 9999. Is that configuration sufficient?

[Ayn]

You should set index and sourcetype on the forwarder. There are somewhat involved ways to do it on-the-fly on the indexer instead, but why would you want to?

[meskildsen]

I didn't succeed in my first try, but guess there is something I might have misunderstood.

1. I have to refer my index on the Universal Forwarder?
2. Do I have to define the sourcetype on the forwarder?
3. I am uncertain of what an where (forwarder / server), to change in the outputs.conf?

BTW. I am also uncertain if it is the correct place I am change the conf files. Where is the preferred location?