In an effort to get our inventory of inputs under control, I'm trying to get all servers to have one place for logs. Eg,
C:\LOGS. When they want to add new files to monitor, they add a directory there, named after the sourcetype:
C:\LOGS\newsourcetype. In that directory, they link (shortcut in the case of Windows) to the actual directory containing their logs. So
I'm having mixed results with this. Sometimes it reads all the logs in the original dir fine and continues to update. Other times it only reads what's there at startup. Once they roll, it stops updating until I restart Splunk again.
EDIT: Clarification, this is for Windows. Symlinks in Linux are working fine using this same layout strategy.
As this seems to be a Windows-only issue, I would suggest replacing shortcuts with an actual NTFS symbolic link. This can be created using one-line in PowerShell as documented here.
# Path is the existing (real) path. Target is the new symlink.
New-Item -ItemType SymbolicLink -Path $path -Target $target
Fun fact, we were already using symlinks. My Windows admin was just using the 'shortcut' term interchangeably. In other words, symlinks in Windows do not behave reliably.
Ahh well, worth a shot. I would open a support case and give them a shot to do some real diagnostics on the issue.
Good point. Testing, will get back soon.
Is OS a factor that you can see for failures? e.g. a POSIX symlink is not the same as a Windows shortcut, and I can imagine there might be an opportunity for bugs to hide in there. Not many are aware that it's actually possible to create a real symlink in Windows, but I would not go down that path unless warranted. (If it seems to be a Windows-specific issue, which you've not stated.)
- any WARN or ERROR entries in the UF logs?
We do the same layout in linux. No issues there. And I understand the links are different. If it had failed entirely, I would have moved on. But it works ... sometimes. I guess that's the same as failure. Sigh. I'll come up with another strategy for Windows. No, the internal logs for file monitoring and tailreader don't give any clues. It sees the files on start-up, but new files are invisible until another restart.
So it is a Windows-only problem? Very interesting. In that case I would try real symbolic links. It's a simple one-liner to do in PowerShell: https://winaero.com/blog/create-symbolic-link-windows-10-powershell/