Getting Data In
Highlighted

IIS WebSIte seperated logging - my config somehow grabs all website files as well. why?

Path Finder

I've got a Win 2008 Web server, and the layout on the disk is as follow:

C:\inetpub\sites\www.fqdn.com\logs\
C:\inetpub\sites\www.fqdn.com\www\
C:\inetpub\sites\another.fqdn.com\logs\
C:\inetpub\sites\another.fqdn.com\www\
etc..

I'm trying to set up a file monitor for C:\inetpub\sites*\logs\ so that only the log files are monitored (and later I want to put on fschange monitoring on the www files (html/php/aspx etc)). I've configured this in search app inputs.conf:

[monitor://C:\inetpub\sites\...\logs\]
sourcetype = iis
disabled = false

Somehow this adds C:\inetpub\sites\*\ to the monitor list, making splunk monitor the www files as well.

What have I missed in my config?

Highlighted

Re: IIS WebSIte seperated logging - my config somehow grabs all website files as well. why?

Influencer

Creating a monitor for the directory C:\inetpub\sites and filtering the files using a whitelist is probably the best option:

[monitor://C:\inetpub\sites\]
sourcetype = iis
disabled = false
whitelist=\\logs\\

View solution in original post

Highlighted

Re: IIS WebSIte seperated logging - my config somehow grabs all website files as well. why?

Path Finder

I will try. Will it work in combination with fschange later when I want to add that to the \www\ folder for tracking changes to the html/php/aspx/js/etc files? Would it be as simple as creating a fschange and a 'whitelist=\www\' to do this?

0 Karma
Highlighted

Re: IIS WebSIte seperated logging - my config somehow grabs all website files as well. why?

Path Finder

It did not work, still lists the \www\ files&folders as well. Do I need a blacklist as well then? Or should it only grab the \logs\ folder and nothing else?

I'm using 4.1.4 btw.

0 Karma
Highlighted

Re: IIS WebSIte seperated logging - my config somehow grabs all website files as well. why?

Path Finder

I do presume that splunk list monitor lists everything thats being monitored...

0 Karma
Highlighted

Re: IIS WebSIte seperated logging - my config somehow grabs all website files as well. why?

Path Finder

By testing this simple regex in RegexBuddy (http://answers.splunk.com/questions/5092/regex-crash-course) with the entire folder structure I have I see whitelist=\\logs\\ is correct (not that I doubted). My question is then, why does splunk list monitor list all the other folders?

0 Karma
Highlighted

Re: IIS WebSIte seperated logging - my config somehow grabs all website files as well. why?

Path Finder

the whitelist doesn't work for my splunk lightforwarder. Even though the simple whitelist should be correct it is not forwarded.

0 Karma
Highlighted

Re: IIS WebSIte seperated logging - my config somehow grabs all website files as well. why?

Path Finder

I have subfolders under \logs.

C:\inetpub\sites\www.fqdn.com\logs\
C:\inetpub\sites\www.fqdn.com\logs\W3SVC4\ (the log files are here - *.log)
C:\inetpub\sites\www.fqdn.com\logs\FTPSVC4\ (log files are here - *.log)
C:\inetpub\sites\www.fqdn.com\www\
C:\inetpub\sites\www.fqdn.com\www\xxx\
C:\inetpub\sites\www.fqdn.com\www\yyy\
C:\inetpub\sites\another.fqdn.com\logs\
C:\inetpub\sites\another.fqdn.com\logs\W3SVC7\
C:\inetpub\sites\another.fqdn.com\logs\FTPSVC7\
C:\inetpub\sites\another.fqdn.com\www\
C:\inetpub\sites\another.fqdn.com\www\ (webfiles files here)
C:\inetpub\sites\another.fqdn.com\www\aaa\ (more webfiles here)
C:\inetpub\sites\another.fqdn.com\www\bbb\ (more webfiles here)

What I want, which I think you guys understand, is to monitor all log files under \logs\, no matter what the fqdn folder name is, and what folders are under \logs.

I confirmed that if I monitor the C:\inetpub\sites\lebara.stag.carrot.no\logs\W3SVC4\ the logs are indexed (as it should), but I can't seem to get the correct config with wildcards etc...

Update:

If i add whitelist=\\logs\\ I get this output from splunk list monitor:

 C:\inetpub\sites\
   C:\inetpub\sites\www.fqdn.com
   C:\inetpub\sites\www.fqdn.com\logs
   C:\inetpub\sites\www.fqdn.com\logs\ProfileWS
   C:\inetpub\sites\www.fqdn.com\logs\W3SVC3
   C:\inetpub\sites\www.fqdn.com\www
   C:\inetpub\sites\www.fqdn.com\www\App_Data
   C:\inetpub\sites\www.fqdn.com\www\aspnet_client
   etc... (for each website)

If I add a '*' after the last backslash - whitelist=\\logs\\* - I get the logfiles in the monitor list at least, but still \www\:

 C:\inetpub\sites\
   C:\inetpub\sites\www.fqdn.com
   C:\inetpub\sites\www.fqdn.com\logs
   C:\inetpub\sites\www.fqdn.com\logs\ProfileWS
   C:\inetpub\sites\www.fqdn.com\logs\ProfileWS\ProfileWS.error.log
   C:\inetpub\sites\www.fqdn.com\logs\ProfileWS\ProfileWS.log
   C:\inetpub\sites\www.fqdn.com\logs\W3SVC3
   C:\inetpub\sites\www.fqdn.com\logs\W3SVC3\u_ex100725.log
   C:\inetpub\sites\www.fqdn.com\logs\W3SVC3\u_ex100726.log
   C:\inetpub\sites\www.fqdn.com\logs\W3SVC3\u_ex100727.log
   C:\inetpub\sites\www.fqdn.com\www
   C:\inetpub\sites\www.fqdn.com\www\App_Data
   C:\inetpub\sites\www.fqdn.com\www\aspnet_client

I'm getting quite frustrated here 😞 And regex is almost like greek to me.

Update 2010-07-29:

I'm now running with the whitelist = \\logs\\ config, but no logs gets sent to the splunk indexer. Checking splunkd.log I see this (and lots of the same kind):

07-29-2010 16:07:59.495 INFO  TailingProcessor - No configurations match, will ignore path='C:\inetpub\sites\www.fqdn.com\www\bin\CIF.xml'.
07-29-2010 16:07:59.511 INFO  TailingProcessor - No configurations match, will ignore path='C:\inetpub\sites\another.fqdn.com\logs\W3SVC7\u_ex100729.log'.
07-29-2010 16:29:20.013 INFO  TailingProcessor - No configurations match, will ignore path='C:\inetpub\sites\www.fqdn.com\www\bin\CIF.xml'.

As you all can see, it doesn't match the \logs\ folder and therefor the logs aren't sent to my indexer... Isn't this strange?

0 Karma
Highlighted

Re: IIS WebSIte seperated logging - my config somehow grabs all website files as well. why?

Path Finder

Finally. This worked:

[monitor://C:\inetpub\sites\]
sourcetype = iis
followTail = 1
whitelist = \\logs\\*
disabled = 0

Note the last star in the whitelist (*)

View solution in original post

0 Karma