topic Re: Issue with UF not uploading files from a specific folder in Monitoring Splunk

Issue with UF not uploading files from a specific folder

Ismail_BSA — Fri, 11 Aug 2023 07:50:48 GMT

Hi,

I have an issue where my UF installed on a linux server is not uploading data to Splunk from a specific folder.

My inputs.conf file contains multiple simliar set ups for several folders to be uploaded. Everything is working perfectly except for one folder. In the inputs.conf file, I have the following set up for this folder:

[monitor:///DATA/remotelogs-ORACLE-MESS/test/*] index=test_bd_oracle sourcetype=test_oracle:audit:xml host_segment = 3

This set up is to upload all files under path /DATA/remotelogs-ORACLE-MESS/test/

However, no files are being uploaded.

What is also wierd is that when I open one of those files using the Linux Vim command, a temporary copy of that file is autocreated with extension .swp and the UF UPLOADS the .swp file.

Any help is appreciated.

Thank you.

Re: Issue with UF not uploading files from a specific folder

scelikok — Thu, 10 Aug 2023 14:28:52 GMT

Hi @Ismail_BSA,

Your other monitor inputs may be hitting the same path, please check all other monitor stanzas if this happens.

Re: Issue with UF not uploading files from a specific folder

Ismail_BSA — Thu, 10 Aug 2023 14:34:33 GMT

Hi @scelikok

Thank you for your reply.

However, I checked all monitor stanzas and they are not hitting the same path, I also renamed the original folder to "test" just to be sure it's unique and not interfearing with other paths.

Re: Issue with UF not uploading files from a specific folder

gcusello — Thu, 10 Aug 2023 15:10:16 GMT

Hi @Ismail_BSA,

the swp extension seems that the file wasn'r correctly closed. you have to manually delete the swp file to avoid the message, this could be the reason but I'm not sure.

Are you seure that there isn't any other copy of the file, eventually with a different name?

Splunk does't read a file twice unless you force double reading with the option "crcSalt = <SOURCE>" in inputs.conf.

Ciao.

Giuseppe

Re: Issue with UF not uploading files from a specific folder

Ismail_BSA — Thu, 10 Aug 2023 15:16:11 GMT

Hi @gcusello

The issue is that splunk is not Uploading my files but only uploading the .swp files (that are autocreated temporelly when I open the files with Vim command).

Are you seure that there isn't any other copy of the file, eventually with a different name?:

The files are recent, I created them my self and I am sure they are not duplicated.

I also tried to copy the entire folder /DATA/remotelogs-ORACLE-MESS/test/ to a new one /DATA/remotelogs-ORACLE-MESS/NEWtest/ and I created a new monitor Stanza but did not resolve my issue (no files are being uploded from the new folder)

[monitor:///DATA/remotelogs-ORACLE-MESS/NEWtest/*]
index=test_bd_oracle
sourcetype=test_oracle:audit:xml
host_segment = 3

I also made sure to restart the Splunk Forwarder on my server.

Re: Issue with UF not uploading files from a specific folder

gcusello — Thu, 10 Aug 2023 15:22:22 GMT

Hi @Ismail_BSA,

as I said, manually delete the swp files.

The issue that copying the files in a new location they aren't read is related to the twice copy.

Ciao.

Giuseppe

Re: Issue with UF not uploading files from a specific folder

Ismail_BSA — Thu, 10 Aug 2023 17:58:46 GMT

Hi @gcusello

I already deleted the .swp files. The new files (.xml) are still not being uploaded

Re: Issue with UF not uploading files from a specific folder

Ismail_BSA — Thu, 10 Aug 2023 20:23:36 GMT

By checking the index="_internal" I found some errors related to my files that are not being uploded, here is an example

-0400 ERROR TailReader [3438 tailreader0] - File will not be read, is too small to match seekptr checksum (file=/file path/). Last time we saw this initcrc, filename was different. You may wish to use larger initCrcLen for this sourcetype, or a CRC salt on this source. Consult the documentation or file a support case online at http://www.splunk.com/page/submit_issue for more info.

I found this other question that seems simlar to my issue

Solved: How do you fix this? ERROR TailingProcessor - File... - Splunk Community

and the resolution was to increase the initCrcLength value

More details could be found on the official doc: inputs.conf - Splunk Documentation

Thank you

Re: Issue with UF not uploading files from a specific folder

gcusello — Fri, 11 Aug 2023 04:48:44 GMT

Hi @Ismail_BSA ,

good for you, see next time!

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated by all the contributors 😉