Getting Data In

How does monitor:// handle windows shortcuts to directories containing logs?

twinspop
Influencer

In an effort to get our inventory of inputs under control, I'm trying to get all servers to have one place for logs. Eg, C:\LOGS. When they want to add new files to monitor, they add a directory there, named after the sourcetype: C:\LOGS\newsourcetype. In that directory, they link (shortcut in the case of Windows) to the actual directory containing their logs. So C:\LOGS\newsourcetype\link1 -> D:\some\path\to\app\log_dir\

I'm having mixed results with this. Sometimes it reads all the logs in the original dir fine and continues to update. Other times it only reads what's there at startup. Once they roll, it stops updating until I restart Splunk again.

EDIT: Clarification, this is for Windows. Symlinks in Linux are working fine using this same layout strategy.

hrottenberg_spl
Splunk Employee
Splunk Employee

As this seems to be a Windows-only issue, I would suggest replacing shortcuts with an actual NTFS symbolic link. This can be created using one-line in PowerShell as documented here.

Example:
# Path is the existing (real) path. Target is the new symlink.
New-Item -ItemType SymbolicLink -Path $path -Target $target

twinspop
Influencer

Fun fact, we were already using symlinks. My Windows admin was just using the 'shortcut' term interchangeably. In other words, symlinks in Windows do not behave reliably.

0 Karma

hrottenberg_spl
Splunk Employee
Splunk Employee

Ahh well, worth a shot. I would open a support case and give them a shot to do some real diagnostics on the issue.

0 Karma

twinspop
Influencer

Good point. Testing, will get back soon.

0 Karma

hrottenberg_spl
Splunk Employee
Splunk Employee

Is OS a factor that you can see for failures? e.g. a POSIX symlink is not the same as a Windows shortcut, and I can imagine there might be an opportunity for bugs to hide in there. Not many are aware that it's actually possible to create a real symlink in Windows, but I would not go down that path unless warranted. (If it seems to be a Windows-specific issue, which you've not stated.)

Other thoughts:
- any WARN or ERROR entries in the UF logs?

twinspop
Influencer

We do the same layout in linux. No issues there. And I understand the links are different. If it had failed entirely, I would have moved on. But it works ... sometimes. I guess that's the same as failure. Sigh. I'll come up with another strategy for Windows. No, the internal logs for file monitoring and tailreader don't give any clues. It sees the files on start-up, but new files are invisible until another restart.

0 Karma

hrottenberg_spl
Splunk Employee
Splunk Employee

So it is a Windows-only problem? Very interesting. In that case I would try real symbolic links. It's a simple one-liner to do in PowerShell: https://winaero.com/blog/create-symbolic-link-windows-10-powershell/