Getting Data In

How can a forwarder monitor a dynamic path?

mawomommoh
Path Finder

How can a forwarder be setup to monitor files with a dynamic path?

For instance, I have a folder structure such as this:
\\shared\tests\{DateTime.NOW}\logs\xxx_yyy_{DateTime.NOW}.xml

DateTime.NOW represents the time which the xml file was generated. There will be multiple {DateTime.NOW} folders in the \\shared\test path.

I have tried some of the solutions stated here: https://answers.splunk.com/answers/33436/monitor-file-with-dynamic-directiory-name.html?utm_source=t...

such as:
\\shared\test\\logs\xxx*
\\shared\test...logs\xxx*
\\shared\test\...\logs\xxx*

but they did not work.

Any help would be appreciated. Thanks!

0 Karma
1 Solution

woodcock
Esteemed Legend

This should definitely work:

[monitor://\\shared\tests\*\logs\xxx_yyy_*.xml]

I suspect that your problem is in the stanza's definition portion, not the file portion.

View solution in original post

woodcock
Esteemed Legend

This should definitely work:

[monitor://\\shared\tests\*\logs\xxx_yyy_*.xml]

I suspect that your problem is in the stanza's definition portion, not the file portion.

mawomommoh
Path Finder

Thanks @woodcock! This works perfectly. I had to restart the forwarder before it worked.

Much appreciated!

0 Karma

FrankVl
Ultra Champion

I would say \\shared\test\...\logs\xxx* should work, unless there is some specific limitation in using that approach for such UNC network share paths.

Have you tried mounting that share on your Splunk server and then pointing Splunk at the mountpoint, rather than using the share path in the inputs.conf?

In general: have you tried monitoring a specific folder, just to determine whether the issue is with the wildcards, or with accessing the share in general?

mawomommoh
Path Finder

Thanks @FrankVI , I have tried monitoring files on the share and it works fine, but like I stated for @MuS 's suggestion above folders get ignored at the ... level of the path.

0 Karma

FrankVl
Ultra Champion

You might want to file a bug report on that then, because theoretically ... should work just as good as * in this case.

Out of curiosity: how long did you give the forwarder time to start reading all the files and folders after making changes to the inputs.conf? I know Splunk can be rather slow at traversing such shared folders and can really take quite some time before discovering all files and starting to read from them.

0 Karma

mawomommoh
Path Finder

I see. Thanks for the insight.

I made the changes and restarted the forwarder, and then waited for 6-8 minutes. Maybe I needed to wait longer.

0 Karma

MuS
SplunkTrust
SplunkTrust

If you are really trying to monitor UNC shares I recommend reading this answer https://answers.splunk.com/answers/218965/how-monitor-logs-on-a-unc-path.html and regarding the wildcarding; this should work \\shared\test\...\logs\xxx*

cheers, MuS

0 Karma

mawomommoh
Path Finder

Thanks @MuS , I tried your suggestion and the forwarder was only able to detect one of the folders in the ... level of the path. It ignores all other folders. And despite detecting this folder only one xml file is forwarded to Splunk.

0 Karma
Get Updates on the Splunk Community!

Last Chance to Submit Your Paper For BSides Splunk - Deadline is August 12th!

Hello everyone! Don't wait to submit - The deadline is August 12th! We have truly missed the community so ...

Ready, Set, SOAR: How Utility Apps Can Up Level Your Playbooks!

 WATCH NOW Powering your capabilities has never been so easy with ready-made Splunk® SOAR Utility Apps. Parse ...

DevSecOps: Why You Should Care and How To Get Started

 WATCH NOW In this Tech Talk we will talk about what people mean by DevSecOps and deep dive into the different ...