Splunk UF ingestion errors - cannot open some files

[NoSpaces]

Hello to everyone!
I have many FlexEngine.log files in different directories that are ingested by Splunk UF 9.0.8
The path from logs is network share on the Windows Server, in which client-side application write via SMB
Some files are ingested without errors, but others have errors that you can see below:

03-18-2024 11:39:23.852 +0300 ERROR TailReader [10000 tailreader0] - error from read call from 'L:\App\UEM\CB\UserSettings\username\FlexEngine.log'.

03-18-2024 11:39:27.839 +0300 WARN  FileClassifierManager [10000 tailreader0] - Unable to open 'L:\App\UEM\CB\UserSettings\username\FlexEngine.log'.

03-18-2024 11:39:27.839 +0300 WARN  FileClassifierManager [10000 tailreader0] - The file 'L:\App\UEM\CB\UserSettings\username\FlexEngine.log' is invalid. Reason: cannot_open.

inputs.conf looks like:

[monitor://L:\App\UEM\CB\UserSettings\*\FlexEngine.log]
disabled = false
index = dem
sourcetype  = dem_file_log

 
and this is an example of a file:

2024-03-18 07:01:32.889 [INFO ] Starting FlexEngine v9.9.0.905 [IFP#14d600e0-T5>>]
2024-03-18 07:01:32.889 [INFO ] Running as Group Policy client-side extension
2024-03-18 07:01:32.889 [INFO ] Performing path-based import
2024-03-18 07:01:32.890 [DEBUG]    User: domain\username, Computer: ComputerName, OS: x64-win10 (Version 1809, BuildNumber 17763.5329, SuiteMask 100, ProductType 1/7d, Lang 0419, IE 11.1790.17763.0, VMware VDM 7.12.0, App Volumes 2.18.6.24, DEM 9.9.0.905, ProcInfo 1/1/2/2, UTC+03:00N), PTS: 6108/2768/1CT
2024-03-18 07:01:32.890 [DEBUG]    Profile state: local (0x00000204)
2024-03-18 07:01:32.890 [DEBUG]    Recursively processing config files from path '\\domain\app\UEM\CB\Settings\general'
2024-03-18 07:01:32.890 [DEBUG]    Using profile archive path '\\domain\app\UEM\CB\UserSettings\username'
2024-03-18 07:01:32.890 [DEBUG]    Last modified dates will be restored
2024-03-18 07:01:32.890 [DEBUG]    Logging to file '\\domain\app\UEM\CB\UserSettings\username\FlexEngine.log'
2024-03-18 07:01:32.890 [DEBUG]    Log file will be overwritten when larger than 512 kilobytes

Which problems can lead to these errors?
Can it be file-blocking by a client-side app, or must Splunk UF handle this situation?

[isoutamo]

Hi

when you want/need to read network shares on window machine, you must install splunk UF to run as domain user not a local. Otherwise it cannot access those files on shares.

• Prepare your Windows network to run Splunk Enterprise as a network or domain user
• https://community.splunk.com/t5/Installation/Domain-Account-for-UF/m-p/523581

r. Ismo

[NoSpaces]

I think that you do not understand me correctly
In my situation, logs ingested from the local disk

[isoutamo]

You said

"The path from logs is network share on the Windows Server, in which client-side application write via SMB".

Are you sure that those files haven't permissions which allow only AD account access those?

[NoSpaces]

Yes, I'm sure
Our Splunk UF instance run using the system account
And problem files also require permission
I attached a permissions example

[PickleRick]

OK. Is L: drive a local device or a network path mounted locally? (that's not clear from your description).

[NoSpaces]

For Splunk UF, it is a local hard drive
For client application, it is a network drive

[PickleRick]

Then I should expect it's as you said - something about file locking. There is another input type for windows which might be able to help here - MonitorNoHandle. But it has quite a few limitations, judging from the spec. And I've never used it so I can't tell you how it performs.

[NoSpaces]

Limitations of MonitorNoHandle are really significant:

<path> must be a fully qualified path name to a specific file. Wildcards
  and directories are not accepted.

In my situation, it means that I need script-made inputs.conf that will contain hundreds of monitors

[isoutamo]

One old post where has presented some kind of workaround

https://community.splunk.com/t5/Monitoring-Splunk/Why-splunkd-cannot-read-input-files-created-in-sou...

Maybe this helps or not?

[NoSpaces]

I think that copying files to another directory will resolve the problem with file blocking (if it really is)
But it is also quite difficult because of the large amount of files and dirs