IIS WebSIte seperated logging - my config somehow grabs all website files as well. why?

[Joffer]

I've got a Win 2008 Web server, and the layout on the disk is as follow:

C:\inetpub\sites\www.fqdn.com\logs\
C:\inetpub\sites\www.fqdn.com\www\
C:\inetpub\sites\another.fqdn.com\logs\
C:\inetpub\sites\another.fqdn.com\www\
etc..

I'm trying to set up a file monitor for C:\inetpub\sites*\logs\ so that only the log files are monitored (and later I want to put on fschange monitoring on the www files (html/php/aspx etc)). I've configured this in search app inputs.conf:

[monitor://C:\inetpub\sites\...\logs\]
sourcetype = iis
disabled = false

Somehow this adds C:\inetpub\sites\*\ to the monitor list, making splunk monitor the www files as well.

What have I missed in my config?

[ziegfried · Accepted solution]

Creating a monitor for the directory C:\inetpub\sites and filtering the files using a whitelist is probably the best option:

[monitor://C:\inetpub\sites\]
sourcetype = iis
disabled = false
whitelist=\\logs\\

[Joffer · Accepted solution]

Finally. This worked:

[monitor://C:\inetpub\sites\]
sourcetype = iis
followTail = 1
whitelist = \\logs\\*
disabled = 0

Note the last star in the whitelist (*)

[Joffer]

I have subfolders under \logs.

C:\inetpub\sites\www.fqdn.com\logs\
C:\inetpub\sites\www.fqdn.com\logs\W3SVC4\ (the log files are here - *.log)
C:\inetpub\sites\www.fqdn.com\logs\FTPSVC4\ (log files are here - *.log)
C:\inetpub\sites\www.fqdn.com\www\
C:\inetpub\sites\www.fqdn.com\www\xxx\
C:\inetpub\sites\www.fqdn.com\www\yyy\
C:\inetpub\sites\another.fqdn.com\logs\
C:\inetpub\sites\another.fqdn.com\logs\W3SVC7\
C:\inetpub\sites\another.fqdn.com\logs\FTPSVC7\
C:\inetpub\sites\another.fqdn.com\www\
C:\inetpub\sites\another.fqdn.com\www\ (webfiles files here)
C:\inetpub\sites\another.fqdn.com\www\aaa\ (more webfiles here)
C:\inetpub\sites\another.fqdn.com\www\bbb\ (more webfiles here)

What I want, which I think you guys understand, is to monitor all log files under \logs\, no matter what the fqdn folder name is, and what folders are under \logs.

I confirmed that if I monitor the C:\inetpub\sites\lebara.stag.carrot.no\logs\W3SVC4\ the logs are indexed (as it should), but I can't seem to get the correct config with wildcards etc...

Update:

If i add whitelist=\\logs\\ I get this output from splunk list monitor:

 C:\inetpub\sites\
   C:\inetpub\sites\www.fqdn.com
   C:\inetpub\sites\www.fqdn.com\logs
   C:\inetpub\sites\www.fqdn.com\logs\ProfileWS
   C:\inetpub\sites\www.fqdn.com\logs\W3SVC3
   C:\inetpub\sites\www.fqdn.com\www
   C:\inetpub\sites\www.fqdn.com\www\App_Data
   C:\inetpub\sites\www.fqdn.com\www\aspnet_client
   etc... (for each website)

If I add a '*' after the last backslash - whitelist=\\logs\\* - I get the logfiles in the monitor list at least, but still \www\:

 C:\inetpub\sites\
   C:\inetpub\sites\www.fqdn.com
   C:\inetpub\sites\www.fqdn.com\logs
   C:\inetpub\sites\www.fqdn.com\logs\ProfileWS
   C:\inetpub\sites\www.fqdn.com\logs\ProfileWS\ProfileWS.error.log
   C:\inetpub\sites\www.fqdn.com\logs\ProfileWS\ProfileWS.log
   C:\inetpub\sites\www.fqdn.com\logs\W3SVC3
   C:\inetpub\sites\www.fqdn.com\logs\W3SVC3\u_ex100725.log
   C:\inetpub\sites\www.fqdn.com\logs\W3SVC3\u_ex100726.log
   C:\inetpub\sites\www.fqdn.com\logs\W3SVC3\u_ex100727.log
   C:\inetpub\sites\www.fqdn.com\www
   C:\inetpub\sites\www.fqdn.com\www\App_Data
   C:\inetpub\sites\www.fqdn.com\www\aspnet_client

I'm getting quite frustrated here 😞 And regex is almost like greek to me.

Update 2010-07-29:

I'm now running with the whitelist = \\logs\\ config, but no logs gets sent to the splunk indexer. Checking splunkd.log I see this (and lots of the same kind):

07-29-2010 16:07:59.495 INFO  TailingProcessor - No configurations match, will ignore path='C:\inetpub\sites\www.fqdn.com\www\bin\CIF.xml'.
07-29-2010 16:07:59.511 INFO  TailingProcessor - No configurations match, will ignore path='C:\inetpub\sites\another.fqdn.com\logs\W3SVC7\u_ex100729.log'.
07-29-2010 16:29:20.013 INFO  TailingProcessor - No configurations match, will ignore path='C:\inetpub\sites\www.fqdn.com\www\bin\CIF.xml'.

As you all can see, it doesn't match the \logs\ folder and therefor the logs aren't sent to my indexer... Isn't this strange?

[ziegfried · Accepted solution]

Creating a monitor for the directory C:\inetpub\sites and filtering the files using a whitelist is probably the best option:

[monitor://C:\inetpub\sites\]
sourcetype = iis
disabled = false
whitelist=\\logs\\

[Joffer]

the whitelist doesn't work for my splunk lightforwarder. Even though the simple whitelist should be correct it is not forwarded.

[Joffer]

By testing this simple regex in RegexBuddy (http://answers.splunk.com/questions/5092/regex-crash-course) with the entire folder structure I have I see whitelist=\\logs\\ is correct (not that I doubted). My question is then, why does splunk list monitor list all the other folders?

[Joffer]

I do presume that splunk list monitor lists everything thats being monitored...

[Joffer]

It did not work, still lists the \www\ files&folders as well. Do I need a blacklist as well then? Or should it only grab the \logs\ folder and nothing else?

I'm using 4.1.4 btw.

[Joffer]

I will try. Will it work in combination with fschange later when I want to add that to the \www\ folder for tracking changes to the html/php/aspx/js/etc files? Would it be as simple as creating a fschange and a 'whitelist=\www\' to do this?