topic Re: How can a forwarder monitor a dynamic path? in Getting Data In

How can a forwarder monitor a dynamic path?

mawomommoh — Thu, 28 Jun 2018 09:00:49 GMT

How can a forwarder be setup to monitor files with a dynamic path?

For instance, I have a folder structure such as this:
\\shared\tests\{DateTime.NOW}\logs\xxx_yyy_{DateTime.NOW}.xml

DateTime.NOW represents the time which the xml file was generated. There will be multiple {DateTime.NOW} folders in the \\shared\test path.

I have tried some of the solutions stated here: https://answers.splunk.com/answers/33436/monitor-file-with-dynamic-directiory-name.html?utm_source=typeahead&utm_medium=newquestion&utm_campaign=no_votes_sort_relev

such as:
\\shared\test\\logs\xxx*
\\shared\test...logs\xxx*
\\shared\test\...\logs\xxx*

but they did not work.

Any help would be appreciated. Thanks!

Re: How can a forwarder monitor a dynamic path?

MuS — Thu, 28 Jun 2018 09:09:22 GMT

If you are really trying to monitor UNC shares I recommend reading this answer https://answers.splunk.com/answers/218965/how-monitor-logs-on-a-unc-path.html and regarding the wildcarding; this should work \\shared\test\...\logs\xxx*

cheers, MuS

Re: How can a forwarder monitor a dynamic path?

FrankVl — Thu, 28 Jun 2018 09:09:26 GMT

I would say \\shared\test\...\logs\xxx* should work, unless there is some specific limitation in using that approach for such UNC network share paths.

Have you tried mounting that share on your Splunk server and then pointing Splunk at the mountpoint, rather than using the share path in the inputs.conf?

In general: have you tried monitoring a specific folder, just to determine whether the issue is with the wildcards, or with accessing the share in general?

Re: How can a forwarder monitor a dynamic path?

woodcock — Sat, 30 Jun 2018 15:33:20 GMT

This should definitely work:

[monitor://\\shared\tests\*\logs\xxx_yyy_*.xml]

I suspect that your problem is in the stanza's definition portion, not the file portion.

Re: How can a forwarder monitor a dynamic path?

mawomommoh — Sun, 01 Jul 2018 06:04:50 GMT

Thanks @MuS , I tried your suggestion and the forwarder was only able to detect one of the folders in the ... level of the path. It ignores all other folders. And despite detecting this folder only one xml file is forwarded to Splunk.

Re: How can a forwarder monitor a dynamic path?

mawomommoh — Sun, 01 Jul 2018 06:07:40 GMT

Thanks @FrankVI , I have tried monitoring files on the share and it works fine, but like I stated for @MuS 's suggestion above folders get ignored at the ... level of the path.

Re: How can a forwarder monitor a dynamic path?

mawomommoh — Sun, 01 Jul 2018 06:09:40 GMT

Thanks @woodcock! This works perfectly. I had to restart the forwarder before it worked.

Much appreciated!

Re: How can a forwarder monitor a dynamic path?

FrankVl — Sun, 01 Jul 2018 08:03:19 GMT

You might want to file a bug report on that then, because theoretically ... should work just as good as * in this case.

Out of curiosity: how long did you give the forwarder time to start reading all the files and folders after making changes to the inputs.conf? I know Splunk can be rather slow at traversing such shared folders and can really take quite some time before discovering all files and starting to read from them.

Re: How can a forwarder monitor a dynamic path?

mawomommoh — Wed, 04 Jul 2018 07:34:02 GMT

I see. Thanks for the insight.

I made the changes and restarted the forwarder, and then waited for 6-8 minutes. Maybe I needed to wait longer.