Getting Data In

How to correctly configure Splunk to monitor a directory with wildcards?

rewritex
Contributor

I must be doing something wrong. Splunk is seeing and indexing the first log file it finds and nothing else after within the same folder and nothing else after in the other folders. Any advice? Thank you. -Sean

Full path to log files are (sact01 - sact15):

/logs/sam/ct/sact01/section01/web
/logs/sam/ct/sact01/sec-02/web
/logs/sam/ct/sact01/sec-03/web
/logs/sam/ct/sact02/section01/web
/logs/sam/ct/sact02/sec-02/web
/logs/sam/ct/sact02/sec-03/web

Log files name format within /web (4 months worth, but i only need 7 days):

request.log.2017-02-28-13-16-04
request.log.2017-03-01-08-12-04
request.log.2017-03-02-13-33-04

Inputs.conf

[monitor:///logs/sam/ct/sact0*/.../web]
_TCP_ROUTING = WestCoast0102
disabled = false
index = test-i007
sourcetype = sam
whitelist = request\.logs*$
crcSalt = <SOURCE>
ignoreOlderThan = 7d

index=_internal source="/opt/splunkforwarder/var/log/splunk/splunkd.log

TailingProcessor - Adding watch on path: /logs/sam/ct.
TailingProcessor - Parsing configuration stanza: monitor:///logs/sam/ct/sactws0*/.../web.
0 Karma
1 Solution

rewritex
Contributor

This is resolved. I had to add the ending /
This works: [monitor:///logs/sam/ct/sact0*/*/web/]

View solution in original post

0 Karma

rewritex
Contributor

This is resolved. I had to add the ending /
This works: [monitor:///logs/sam/ct/sact0*/*/web/]

0 Karma

rewritex
Contributor

This is resolved. I had to add the ending /
This works: [monitor:///logs/sam/ct/sact0*/*/web/]

0 Karma

woodcock
Esteemed Legend

Yes, otherwise it thinks that you are specifying a file name, not a directory.

0 Karma

rewritex
Contributor

I corrected the whitelist = request.logs*$ to whitelist = request.log*$ to match the naming convention.
Still no luck

0 Karma

ddrillic
Ultra Champion

-- Log files name format within /web (4 months worth, but i only need 7 days):

That goes to best practices of the log directories and archiving. From Splunk perspective, it would be much better if this directory structure contains *only * the 7 days of data. The rest of the files should be moved to an archive area. If possible.

Sifting through a large directory structure and identifying the files to monitor is expensive and the software doesn't give you enough indications where it stands when it processes this directory, so we are in the dark.

0 Karma

woodcock
Esteemed Legend

You have told it to search for infinite depth and if you have many subdirectories, Splunk can get WAY too busy. Change it to this:

 [monitor:///logs/sam/ct/sact0*/*/web]
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...