topic Re: Splunk UF ingestion errors - cannot open some files in Getting Data In

Splunk UF ingestion errors - cannot open some files

NoSpaces — Mon, 18 Mar 2024 10:01:43 GMT

Hello to everyone!
I have many FlexEngine.log files in different directories that are ingested by Splunk UF 9.0.8
The path from logs is network share on the Windows Server, in which client-side application write via SMB
Some files are ingested without errors, but others have errors that you can see below:

03-18-2024 11:39:23.852 +0300 ERROR TailReader [10000 tailreader0] - error from read call from 'L:\App\UEM\CB\UserSettings\username\FlexEngine.log'. 03-18-2024 11:39:27.839 +0300 WARN FileClassifierManager [10000 tailreader0] - Unable to open 'L:\App\UEM\CB\UserSettings\username\FlexEngine.log'. 03-18-2024 11:39:27.839 +0300 WARN FileClassifierManager [10000 tailreader0] - The file 'L:\App\UEM\CB\UserSettings\username\FlexEngine.log' is invalid. Reason: cannot_open.

inputs.conf looks like:

[monitor://L:\App\UEM\CB\UserSettings\*\FlexEngine.log] disabled = false index = dem sourcetype = dem_file_log

and this is an example of a file:

2024-03-18 07:01:32.889 [INFO ] Starting FlexEngine v9.9.0.905 [IFP#14d600e0-T5>>] 2024-03-18 07:01:32.889 [INFO ] Running as Group Policy client-side extension 2024-03-18 07:01:32.889 [INFO ] Performing path-based import 2024-03-18 07:01:32.890 [DEBUG] User: domain\username, Computer: ComputerName, OS: x64-win10 (Version 1809, BuildNumber 17763.5329, SuiteMask 100, ProductType 1/7d, Lang 0419, IE 11.1790.17763.0, VMware VDM 7.12.0, App Volumes 2.18.6.24, DEM 9.9.0.905, ProcInfo 1/1/2/2, UTC+03:00N), PTS: 6108/2768/1CT 2024-03-18 07:01:32.890 [DEBUG] Profile state: local (0x00000204) 2024-03-18 07:01:32.890 [DEBUG] Recursively processing config files from path '\\domain\app\UEM\CB\Settings\general' 2024-03-18 07:01:32.890 [DEBUG] Using profile archive path '\\domain\app\UEM\CB\UserSettings\username' 2024-03-18 07:01:32.890 [DEBUG] Last modified dates will be restored 2024-03-18 07:01:32.890 [DEBUG] Logging to file '\\domain\app\UEM\CB\UserSettings\username\FlexEngine.log' 2024-03-18 07:01:32.890 [DEBUG] Log file will be overwritten when larger than 512 kilobytes

Which problems can lead to these errors?
Can it be file-blocking by a client-side app, or must Splunk UF handle this situation?

Re: Splunk UF ingestion errors - cannot open some files

isoutamo — Mon, 18 Mar 2024 11:20:12 GMT

when you want/need to read network shares on window machine, you must install splunk UF to run as domain user not a local. Otherwise it cannot access those files on shares.

r. Ismo

Re: Splunk UF ingestion errors - cannot open some files

NoSpaces — Mon, 18 Mar 2024 11:49:53 GMT

I think that you do not understand me correctly
In my situation, logs ingested from the local disk

Re: Splunk UF ingestion errors - cannot open some files

isoutamo — Mon, 18 Mar 2024 13:57:15 GMT

You said

"The path from logs is network share on the Windows Server, in which client-side application write via SMB".

Are you sure that those files haven't permissions which allow only AD account access those?

Re: Splunk UF ingestion errors - cannot open some files

NoSpaces — Mon, 18 Mar 2024 15:02:14 GMT

Yes, I'm sure
Our Splunk UF instance run using the system account
And problem files also require permission
I attached a permissions example

Re: Splunk UF ingestion errors - cannot open some files

PickleRick — Mon, 18 Mar 2024 15:06:33 GMT

OK. Is L: drive a local device or a network path mounted locally? (that's not clear from your description).

Re: Splunk UF ingestion errors - cannot open some files

NoSpaces — Tue, 19 Mar 2024 13:14:29 GMT

For Splunk UF, it is a local hard drive
For client application, it is a network drive

Re: Splunk UF ingestion errors - cannot open some files

PickleRick — Tue, 19 Mar 2024 20:45:33 GMT

Then I should expect it's as you said - something about file locking. There is another input type for windows which might be able to help here - MonitorNoHandle. But it has quite a few limitations, judging from the spec. And I've never used it so I can't tell you how it performs.

Re: Splunk UF ingestion errors - cannot open some files

NoSpaces — Wed, 20 Mar 2024 06:32:02 GMT

Limitations of MonitorNoHandle are really significant:

<path> must be a fully qualified path name to a specific file. Wildcards
  and directories are not accepted.

In my situation, it means that I need script-made inputs.conf that will contain hundreds of monitors

Re: Splunk UF ingestion errors - cannot open some files

isoutamo — Wed, 20 Mar 2024 09:28:22 GMT

One old post where has presented some kind of workaround

https://community.splunk.com/t5/Monitoring-Splunk/Why-splunkd-cannot-read-input-files-created-in-source-folder/m-p/184598

Maybe this helps or not?

Re: Splunk UF ingestion errors - cannot open some files

NoSpaces — Wed, 20 Mar 2024 12:24:17 GMT

I think that copying files to another directory will resolve the problem with file blocking (if it really is)
But it is also quite difficult because of the large amount of files and dirs