How can a forwarder monitor a dynamic path?

[mawomommoh]

How can a forwarder be setup to monitor files with a dynamic path?

For instance, I have a folder structure such as this:
\\shared\tests\{DateTime.NOW}\logs\xxx_yyy_{DateTime.NOW}.xml

DateTime.NOW represents the time which the xml file was generated. There will be multiple {DateTime.NOW} folders in the \\shared\test path.

I have tried some of the solutions stated here: https://answers.splunk.com/answers/33436/monitor-file-with-dynamic-directiory-name.html?utm_source=t...

such as:
\\shared\test\\logs\xxx*
\\shared\test...logs\xxx*
\\shared\test\...\logs\xxx*

but they did not work.

Any help would be appreciated. Thanks!

[woodcock · Accepted solution]

This should definitely work:

[monitor://\\shared\tests\*\logs\xxx_yyy_*.xml]

I suspect that your problem is in the stanza's definition portion, not the file portion.

[mawomommoh]

Thanks @woodcock! This works perfectly. I had to restart the forwarder before it worked.

Much appreciated!

[FrankVl]

I would say \\shared\test\...\logs\xxx* should work, unless there is some specific limitation in using that approach for such UNC network share paths.

Have you tried mounting that share on your Splunk server and then pointing Splunk at the mountpoint, rather than using the share path in the inputs.conf?

In general: have you tried monitoring a specific folder, just to determine whether the issue is with the wildcards, or with accessing the share in general?

[mawomommoh]

Thanks @FrankVI , I have tried monitoring files on the share and it works fine, but like I stated for @MuS 's suggestion above folders get ignored at the ... level of the path.

[FrankVl]

You might want to file a bug report on that then, because theoretically ... should work just as good as * in this case.

Out of curiosity: how long did you give the forwarder time to start reading all the files and folders after making changes to the inputs.conf? I know Splunk can be rather slow at traversing such shared folders and can really take quite some time before discovering all files and starting to read from them.

[mawomommoh]

I see. Thanks for the insight.

I made the changes and restarted the forwarder, and then waited for 6-8 minutes. Maybe I needed to wait longer.

[MuS]

If you are really trying to monitor UNC shares I recommend reading this answer https://answers.splunk.com/answers/218965/how-monitor-logs-on-a-unc-path.html and regarding the wildcarding; this should work \\shared\test\...\logs\xxx*

cheers, MuS

[mawomommoh]

Thanks @MuS , I tried your suggestion and the forwarder was only able to detect one of the folders in the ... level of the path. It ignores all other folders. And despite detecting this folder only one xml file is forwarded to Splunk.