Getting Data In
Highlighted

Splunk Monitor File and Folder: Not uploading all files

Communicator

HI All,

I am trying to monitor 3 CSVs from a same folder via Splunk : Settings -> Data Input -> Files & Directories.
My Files are Alpha1.csv, Beta1.csv and Gamma1.csv. and the sourcetpyes are Alpha , Beta and Gamma with a proper timestamp (predefined).
so in the Settings -> Data Input -> Files & Directories: I added the files as:-

Full path: C:\FileLOG\Alpha*.csv 
Whitelist:  Alpha
SourceType: Alpha
Index = Index1

Full path: C:\FileLOG\Beta*.csv 
Whitelist:  Beta
SourceType: Beta
Index = Index1

Full path: C:\FileLOG\Gamma*.csv 
Whitelist:  Gamma
SourceType: Gamma
Index = Index1

However, when I check Splunk, it shows me events only from C:\FileLOG\Gamma*.csv. Normal Upload of data is working fine and there is no issue with source type or index. Just the monitor function won't upload all files, although different rules are applied.

thanks for the help and my apologies for such a long post.

Highlighted

Re: Splunk Monitor File and Folder: Not uploading all files

Ultra Champion

Can you share a screenshot of the relevant part of the Settings -> Data Input -> Files & Directories page, showing the entries for those 3 inputs? (or if you know how to find it: share the inputs.conf file that was generated for this).

0 Karma
Highlighted

Re: Splunk Monitor File and Folder: Not uploading all files

Communicator

I cannot share the screenshot from Splunk (Sensitive Data), but I can share entries in input.conf in

C:\Program Files\Splunk\etc\system\local
[default]
host = CPX-XXXXXXXX

0 Karma
Highlighted

Re: Splunk Monitor File and Folder: Not uploading all files

Ultra Champion

That's not the relevant inputs.conf file. Not sure in which app the config was added, but you might want to take a look in etc/apps/search/local/inputs.conf

0 Karma
Highlighted

Re: Splunk Monitor File and Folder: Not uploading all files

SplunkTrust
SplunkTrust

check the results of this search for crcinit issues, permissions isssues, etc.

index=_internal log_level=warn* OR log_level=error

View solution in original post

Highlighted

Re: Splunk Monitor File and Folder: Not uploading all files

Communicator

Thanks for it. I found the error as:

06-14-2018 10:43:10.159 +0200 ERROR TailReader - File will not be read, is too small to match seekptr checksum (file=C:\FileLOG\Alpha1.csv). Last time we saw this initcrc, filename was different. You may wish to use larger initCrcLen for this sourcetype, or a CRC salt on this source. Consult the documentation or file a support case online at http://www.splunk.com/page/submit_issue for more info.

06-14-2018 10:43:10.079 +0200 ERROR TailReader - File will not be read, seekptr checksum did not match (file=C:\FileLOG\Beta1.csv ). Last time we saw this initcrc, filename was different. You may wish to use larger initCrcLen for this sourcetype, or a CRC salt on this source. Consult the documentation or file a support case online at http://www.splunk.com/page/submit_issue for more info.

Highlighted

Re: Splunk Monitor File and Folder: Not uploading all files

Ultra Champion

Sounds like the first bits of those CSV files may be too similar. Increasing the initCrcLen or adding crcSalt = <SOURCE> (literally like that) in the inputs.conf for each of those inputs might resolve your issue.

0 Karma
Highlighted

Re: Splunk Monitor File and Folder: Not uploading all files

Communicator

Thanks a lot for the comment. Now my input.conf looks like:

[default]
host = CPX-XXXXXXXX
crcSalt =

Do I need to specify all files and add crcSalt for all of them? Sorry but the Splunk answer is removing the SOURCE keyword automatically 🙂

0 Karma
Highlighted

Re: Splunk Monitor File and Folder: Not uploading all files

Ultra Champion

No, you need to find the inputs.conf file where the inputs are already configured (as mentioned in my other comment, my guess would be they are under etc/apps/search/local/). And then add the crcSalt setting there in each section for the three csv inputs.

0 Karma
Highlighted

Re: Splunk Monitor File and Folder: Not uploading all files

Communicator

Sorry but there is no input.conf under etc/apps/search/local/. I found one under Splunk\etc\system\local

but there is no different lines for different sourcetype. I believe I am not looking at the correct place. 😞

0 Karma