topic Re: Wildcards in paths of inputs.conf in Getting Data In

Wildcards in paths of inputs.conf

echalex — Tue, 24 Apr 2012 12:40:59 GMT

Hi,

I must confess I'm still not understanding how wildcards work in inputs.conf. I've got a clustered application, with five instances on one server. The instances are named live-1,live-2,live-3,staging-1,staging-2. They're all located in /opt/INSTANCE_NAME

I'm trying to monitor all the live instances

[monitor:///opt/foo/live-*/logs/]
index = foo_live
sourcetype = log4j
crcSalt = <SOURCE>
blacklist= (\.(gz|bz2|z|zip)$)

The problem with this stanza is that everything under /opt/foo will be listed by 'splunk list monitor'. More than 16000 files... Including everything in /opt/foo/staging-[12] and /opt/foo/whatever. I don't understand how that's possible, since none of those paths include the "live-" part, but anyway...

Problem number 2 is that nothing will actually be forwarded to the indexer by this.

The documentation seems pretty straigh-forward on this, so I really don't understand why it isn't working. If I list every directory as individual stanzas, the forwarding will work as expected. However, I would really need a generic solution to match all future environments as well. (Several applications, several instances.)

Re: Wildcards in paths of inputs.conf

rgcurry — Tue, 24 Apr 2012 17:40:16 GMT

Are all of the logs named the same for each instance in their own directory? Maybe you can try this if that is so:

[monitor:///opt/foo/*]

index = (?:live-[1-5]{1}/logs/foo_live.log)$
sourcetype = log4j

crcSalt =

blacklist= (.(gz|bz2|z|zip)$)

I've never put a RegEx expression in the monitor stanza and not certain that would work.

Re: Wildcards in paths of inputs.conf

echalex — Wed, 25 Apr 2012 07:00:17 GMT

rgcurry,
Yes, to a degree. The application is running under Tomcat, so there are files named catalina.out and files with the date pattern foo.yyyy-mm-dd.log.

Not sure what you're trying to do with the regex in index, though... (?)

Re: Wildcards in paths of inputs.conf

echalex — Mon, 20 Aug 2012 11:59:32 GMT

Replying to myself, in case someone has an interest in the answer:

The problem with splunk list monitor was encountered in a Universal Forwarder version 4.2.3.

This seems to have been fixed in 4.3.1 (or in between). I found this out by first comparing the output of a 4.2.3 and 4.3.1 Universal forwarder, then confirmed by upgrading the 4.2.3 SUF to 4.3.1.

In other word, splunk list monitor will only list the logs matching the monitor-stanza.