Getting Data In

Wildcards in paths of inputs.conf

Builder

Hi,

I must confess I'm still not understanding how wildcards work in inputs.conf. I've got a clustered application, with five instances on one server. The instances are named live-1,live-2,live-3,staging-1,staging-2. They're all located in /opt/INSTANCE_NAME

I'm trying to monitor all the live instances

[monitor:///opt/foo/live-*/logs/]
index = foo_live
sourcetype = log4j
crcSalt = <SOURCE>
blacklist= (\.(gz|bz2|z|zip)$)

The problem with this stanza is that everything under /opt/foo will be listed by 'splunk list monitor'. More than 16000 files... Including everything in /opt/foo/staging-[12] and /opt/foo/whatever. I don't understand how that's possible, since none of those paths include the "live-" part, but anyway...

Problem number 2 is that nothing will actually be forwarded to the indexer by this.

The documentation seems pretty straigh-forward on this, so I really don't understand why it isn't working. If I list every directory as individual stanzas, the forwarding will work as expected. However, I would really need a generic solution to match all future environments as well. (Several applications, several instances.)

0 Karma
1 Solution

Builder

Replying to myself, in case someone has an interest in the answer:

The problem with splunk list monitor was encountered in a Universal Forwarder version 4.2.3.

This seems to have been fixed in 4.3.1 (or in between). I found this out by first comparing the output of a 4.2.3 and 4.3.1 Universal forwarder, then confirmed by upgrading the 4.2.3 SUF to 4.3.1.

In other word, splunk list monitor will only list the logs matching the monitor-stanza.

View solution in original post

0 Karma

Builder

Replying to myself, in case someone has an interest in the answer:

The problem with splunk list monitor was encountered in a Universal Forwarder version 4.2.3.

This seems to have been fixed in 4.3.1 (or in between). I found this out by first comparing the output of a 4.2.3 and 4.3.1 Universal forwarder, then confirmed by upgrading the 4.2.3 SUF to 4.3.1.

In other word, splunk list monitor will only list the logs matching the monitor-stanza.

View solution in original post

0 Karma

Contributor

Are all of the logs named the same for each instance in their own directory? Maybe you can try this if that is so:

[monitor:///opt/foo/*]

index = (?:live-[1-5]{1}/logs/foo_live.log)$
sourcetype = log4j

crcSalt =

blacklist= (.(gz|bz2|z|zip)$)

I've never put a RegEx expression in the monitor stanza and not certain that would work.

Builder

rgcurry,
Yes, to a degree. The application is running under Tomcat, so there are files named catalina.out and files with the date pattern foo.yyyy-mm-dd.log.

Not sure what you're trying to do with the regex in index, though... (?)

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!