topic Re: Inputs.conf with wildcards in Getting Data In

Inputs.conf with wildcards

peter_gianusso — Thu, 13 Sep 2012 23:30:18 GMT

[monitor://\\njros1bva0597\d$\LogFiles\W3SVC1\*.log]
disabled = false
host = NJROS1BVA0621
alwaysOpenFile = 1
sourcetype = Image Importer Logs

With the stanza above, splunk reports the number of files is 3. There are only 2 files in the directory!! Also we can get it to actual index any of the files in the directory. There are only 2 files in the directory and they both end in .log

Please help

Re: Inputs.conf with wildcards

peter_gianusso — Thu, 13 Sep 2012 23:30:50 GMT

This is the stanza

[monitor://\njros1bva0597\d$\LogFiles\W3SVC1*.log]
disabled = false
host = NJROS1BVA0621
alwaysOpenFile = 1
sourcetype = Image Importer Logs

Re: Inputs.conf with wildcards

peter_gianusso — Thu, 13 Sep 2012 23:32:02 GMT

for some reason the splunkbase is removing the backslashes in the path!!! Bottom line we want all files ending in .log in the W3SVC1 directory

Re: Inputs.conf with wildcards

peter_gianusso — Thu, 13 Sep 2012 23:33:57 GMT

we cannot use whitelisting due to another splunk limitation

Re: Inputs.conf with wildcards

cmonig — Fri, 14 Sep 2012 09:19:52 GMT

Hi,

I'm not sure if this is really necessary, but don't you have to escape the backslashes in the monitor stanza?

[monitor://\\njros1bva0597\\d$\\LogFiles\\W3SVC1\\*.log]

Also, you could try to remove the dot after the wildcard, and see if that helps:

[monitor://\\njros1bva0597\\d$\\LogFiles\\W3SVC1\\*log]

Good luck!
:-)

Cheers,

Christoph

Re: Inputs.conf with wildcards

peter_gianusso — Fri, 14 Sep 2012 14:03:07 GMT

tried this
monitor://\njros1bva0597\d$\LogFiles\W3SVC1\*log

I think the problem is our share d$...i think when everything get converted to regex the dollar sign messes things up but we don't know to fix

Re: Inputs.conf with wildcards

aholzer — Fri, 14 Sep 2012 14:09:22 GMT

try escaping the dollar sign: \$

Re: Inputs.conf with wildcards

peter_gianusso — Fri, 14 Sep 2012 18:28:48 GMT

escaping the $ did not work...didn't even recognize the input

went back to my original config that was in the question. Here's some more info from the tailing status

parent \njros1bva0597\d$\LogFiles\W3SVC1*.log
type File did not match whitelist '^\\njros1bva0597\d$\LogFiles\W3SVC1\[^\]*.log$'.

Re: Inputs.conf with wildcards

aholzer — Fri, 14 Sep 2012 18:41:28 GMT

Have you read these two similar questions:
http://splunk-base.splunk.com/answers/2775/regexs-and-windows-paths-in-inputsconf-and-propsconf
&
http://splunk-base.splunk.com/answers/26094/whats-the-syntax-for-monitoring-a-local-windows-directory-or-file

It probably gets really tricky when you are trying to use the administrative pathing to a different box. Have you considered installing a universal forwarder directly on the box you are trying to monitor files from?

Re: Inputs.conf with wildcards

lguinn2 — Fri, 14 Sep 2012 18:48:22 GMT

From the manual: "Warning: In Windows, you cannot currently use a wildcard at the root level." Which is what you are doing. And to answer cmonig, I believe that the double-backslash \\ is only required in whitelists and blacklists, not the monitor stanza itself. (And you don't necessarily need a backslash in a whitelist, as it is a regular expression.)

The following should fix the problem:

[monitor://\\njros1bva0597\d$\LogFiles\W3SVC1]
disabled = false
host = NJROS1BVA0621
alwaysOpenFile = 1
sourcetype = Image Importer Logs
whitelist=.log$

I don't understand why you can't use whitelists. If you can't, then you are stuck with monitoring everything in the directory or finding another way around this AFAIK.

I would personally not use a sourcetype with spaces, but that's up to you. Also, the number of files that Splunk is monitoring may also include directories in the path, I think. To see what is actually being monitored, try this

$SPLUNK_HOME\bin\splunk list monitor

And Splunk will give you a list of the files it is monitoring - it may be a longer list than you expect!

Good move on checking the internal logs for the tailing status.

Updated: you said in the comments: "I want the ROUTEDB log files to get one source type and UPDATEDB to get another source type. I don't want or need the remaining 10 files." You can override the sourcetype setting as needed in props.conf - more info here in the middle of the page at Specify sourcetype for source. Perhaps this would help you so that you can use a whitelist and not violate the rule about multiple stanzas with the same path.

If the ultimate problem is the $ in your path name, can you create a link and use it instead? For example, create a link to the d$ directory named d.dollar, and then make the monitor stanza look like this:

[monitor://\\njros1bva0597\d.dollar\LogFiles\W3SVC1]

Re: Inputs.conf with wildcards

peter_gianusso — Fri, 14 Sep 2012 18:56:23 GMT

I don't think I can use whitelists because the Splunk SE guys told me that I couldn't.

In the end, I have 20 files in the directory. 5 that are named UPDATEDB-MM-DD-YY.log and 5 that are named ROUTEDB-MM-DD-YY.log. The remaining 10 are text files and binaries.

I want the ROUTEDB log files to get one source type and UPDATEDB to get another source type. I don't want or need the remaining 10 files.

Re: Inputs.conf with wildcards

peter_gianusso — Fri, 14 Sep 2012 18:56:45 GMT

First, they told me to use whitelists. But then, based on my experience, you can't have more than 1 monitor with the same path. Then they told me to use wildcards. I have been trying to get this simple example going before I tackle the real problem.

I have confirmed that the $ is the issue. I just don't know how to fix it.

monitor://\njros1bva0597\dshare\LogFiles\W3SVC1*.log
or
monitor://\njros1bva0597\dshare\LogFiles\W3SVC1\UPDATEDB*.log

All work just great.

It's our standard of creating shares like D$ that seems to be the issue.

Re: Inputs.conf with wildcards

lguinn2 — Fri, 14 Sep 2012 19:33:21 GMT

You are correct - you can't have more than one monitor on the same path. But if there are only text files and binaries in the directory, along with the log files - why would you have more than one monitor stanza with the same path?

Re: Inputs.conf with wildcards

peter_gianusso — Fri, 14 Sep 2012 19:51:21 GMT

Thanks. I spoke with them again and we are going with the props.conf file. They acknowledged that splunk struggles with the $ in the path when you have a wildcard as well.