Getting Data In

How to correctly configure Splunk to monitor a directory with wildcards?

Contributor

I must be doing something wrong. Splunk is seeing and indexing the first log file it finds and nothing else after within the same folder and nothing else after in the other folders. Any advice? Thank you. -Sean

Full path to log files are (sact01 - sact15):

/logs/sam/ct/sact01/section01/web
/logs/sam/ct/sact01/sec-02/web
/logs/sam/ct/sact01/sec-03/web
/logs/sam/ct/sact02/section01/web
/logs/sam/ct/sact02/sec-02/web
/logs/sam/ct/sact02/sec-03/web

Log files name format within /web (4 months worth, but i only need 7 days):

request.log.2017-02-28-13-16-04
request.log.2017-03-01-08-12-04
request.log.2017-03-02-13-33-04

Inputs.conf

[monitor:///logs/sam/ct/sact0*/.../web]
_TCP_ROUTING = WestCoast0102
disabled = false
index = test-i007
sourcetype = sam
whitelist = request\.logs*$
crcSalt = <SOURCE>
ignoreOlderThan = 7d

index=_internal source="/opt/splunkforwarder/var/log/splunk/splunkd.log

TailingProcessor - Adding watch on path: /logs/sam/ct.
TailingProcessor - Parsing configuration stanza: monitor:///logs/sam/ct/sactws0*/.../web.
0 Karma
1 Solution

Contributor

This is resolved. I had to add the ending /
This works: [monitor:///logs/sam/ct/sact0*/*/web/]

View solution in original post

0 Karma

Contributor

This is resolved. I had to add the ending /
This works: [monitor:///logs/sam/ct/sact0*/*/web/]

View solution in original post

0 Karma

Contributor

This is resolved. I had to add the ending /
This works: [monitor:///logs/sam/ct/sact0*/*/web/]

0 Karma

Esteemed Legend

Yes, otherwise it thinks that you are specifying a file name, not a directory.

0 Karma

Contributor

I corrected the whitelist = request.logs*$ to whitelist = request.log*$ to match the naming convention.
Still no luck

0 Karma

Ultra Champion

-- Log files name format within /web (4 months worth, but i only need 7 days):

That goes to best practices of the log directories and archiving. From Splunk perspective, it would be much better if this directory structure contains *only * the 7 days of data. The rest of the files should be moved to an archive area. If possible.

Sifting through a large directory structure and identifying the files to monitor is expensive and the software doesn't give you enough indications where it stands when it processes this directory, so we are in the dark.

0 Karma

Esteemed Legend

You have told it to search for infinite depth and if you have many subdirectories, Splunk can get WAY too busy. Change it to this:

 [monitor:///logs/sam/ct/sact0*/*/web]
0 Karma
Don’t Miss Global Splunk
User Groups Week!

Free LIVE events worldwide 2/8-2/12
Connect, learn, and collect rad prizes and swag!