topic Splunk Monitor File and Folder: Not uploading all files in Getting Data In

Splunk Monitor File and Folder: Not uploading all files

Chandras11 — Thu, 14 Jun 2018 10:45:23 GMT

HI All,

I am trying to monitor 3 CSVs from a same folder via Splunk : Settings -> Data Input -> Files & Directories.
My Files are Alpha1.csv, Beta1.csv and Gamma1.csv. and the sourcetpyes are Alpha , Beta and Gamma with a proper timestamp (predefined).
so in the Settings -> Data Input -> Files & Directories: I added the files as:-

Full path: C:\FileLOG\Alpha*.csv 
Whitelist:  Alpha
SourceType: Alpha
Index = Index1

Full path: C:\FileLOG\Beta*.csv 
Whitelist:  Beta
SourceType: Beta
Index = Index1

Full path: C:\FileLOG\Gamma*.csv 
Whitelist:  Gamma
SourceType: Gamma
Index = Index1

However, when I check Splunk, it shows me events only from C:\FileLOG\Gamma*.csv. Normal Upload of data is working fine and there is no issue with source type or index. Just the monitor function won't upload all files, although different rules are applied.

thanks for the help and my apologies for such a long post.

Re: Splunk Monitor File and Folder: Not uploading all files

FrankVl — Thu, 14 Jun 2018 11:18:36 GMT

Can you share a screenshot of the relevant part of the Settings -> Data Input -> Files & Directories page, showing the entries for those 3 inputs? (or if you know how to find it: share the inputs.conf file that was generated for this).

Re: Splunk Monitor File and Folder: Not uploading all files

Chandras11 — Thu, 14 Jun 2018 11:25:08 GMT

I cannot share the screenshot from Splunk (Sensitive Data), but I can share entries in input.conf in

C:\Program Files\Splunk\etc\system\local
[default]
host = CPX-XXXXXXXX

Re: Splunk Monitor File and Folder: Not uploading all files

jkat54 — Thu, 14 Jun 2018 11:25:11 GMT

check the results of this search for crcinit issues, permissions isssues, etc.

index=_internal log_level=warn* OR log_level=error

Re: Splunk Monitor File and Folder: Not uploading all files

Chandras11 — Thu, 14 Jun 2018 11:29:56 GMT

Thanks for it. I found the error as:

06-14-2018 10:43:10.159 +0200 ERROR TailReader - File will not be read, is too small to match seekptr checksum (file=C:\FileLOG\Alpha1.csv). Last time we saw this initcrc, filename was different. You may wish to use larger initCrcLen for this sourcetype, or a CRC salt on this source. Consult the documentation or file a support case online at http://www.splunk.com/page/submit_issue for more info.

06-14-2018 10:43:10.079 +0200 ERROR TailReader - File will not be read, seekptr checksum did not match (file=C:\FileLOG\Beta1.csv ). Last time we saw this initcrc, filename was different. You may wish to use larger initCrcLen for this sourcetype, or a CRC salt on this source. Consult the documentation or file a support case online at http://www.splunk.com/page/submit_issue for more info.

Re: Splunk Monitor File and Folder: Not uploading all files

FrankVl — Thu, 14 Jun 2018 11:36:45 GMT

Sounds like the first bits of those CSV files may be too similar. Increasing the initCrcLen or adding crcSalt = <SOURCE> (literally like that) in the inputs.conf for each of those inputs might resolve your issue.

Re: Splunk Monitor File and Folder: Not uploading all files

FrankVl — Thu, 14 Jun 2018 11:37:36 GMT

That's not the relevant inputs.conf file. Not sure in which app the config was added, but you might want to take a look in etc/apps/search/local/inputs.conf

Re: Splunk Monitor File and Folder: Not uploading all files

Chandras11 — Thu, 14 Jun 2018 11:42:35 GMT

Thanks a lot for the comment. Now my input.conf looks like:

[default]
host = CPX-XXXXXXXX
crcSalt =

Do I need to specify all files and add crcSalt for all of them? Sorry but the Splunk answer is removing the SOURCE keyword automatically 🙂

Re: Splunk Monitor File and Folder: Not uploading all files

FrankVl — Thu, 14 Jun 2018 11:44:56 GMT

No, you need to find the inputs.conf file where the inputs are already configured (as mentioned in my other comment, my guess would be they are under etc/apps/search/local/). And then add the crcSalt setting there in each section for the three csv inputs.

Re: Splunk Monitor File and Folder: Not uploading all files

Chandras11 — Thu, 14 Jun 2018 11:50:58 GMT

Sorry but there is no input.conf under etc/apps/search/local/. I found one under Splunk\etc\system\local

but there is no different lines for different sourcetype. I believe I am not looking at the correct place. 😞

Re: Splunk Monitor File and Folder: Not uploading all files

FrankVl — Thu, 14 Jun 2018 12:04:36 GMT

Have a look on the data inputs page that lists all the configured inputs. That should also mention in which app the config was stored. That should help you find it on the filesystem.

Re: Splunk Monitor File and Folder: Not uploading all files

Chandras11 — Thu, 14 Jun 2018 12:29:01 GMT

Thanks a lot Frank. I got it with the command: $ find . -name inputs.conf -print . It was udner the C:\Program Files\Splunk\etc\apps\MYAPP\local folder

Re: Splunk Monitor File and Folder: Not uploading all files

jkat54 — Thu, 14 Jun 2018 22:59:06 GMT

So now you will have a stanza like this:

 [monitor::/path/to/folder/*.log]
 crcSalt=<SOURCE>

And that will fix the problem after restarting.

You may have to clear the fish bucket or use another index name to get it to reload the data.

Re: Splunk Monitor File and Folder: Not uploading all files

Chandras11 — Tue, 29 Sep 2020 20:00:37 GMT

So, if anyone gets the same issue, the workaround is:-
First run the query in splunk search head:-
index=_internal log_level=warn* OR log_level=error and check for the time, when you put the CSVs in the folder.
You will get the possible error or warning. I got the following:
06-14-2018 10:43:10.079 +0200 ERROR TailReader - File will not be read, seekptr checksum did not match (file=C:\FileLOG\Beta1.csv ). Last time we saw this initcrc, filename was different. You may wish to use larger initCrcLen for this sourcetype, or a CRC salt on this source. Consult the documentation or file a support case online at http://www.splunk.com/page/submit_issue for more info.

So I had the issue with crcSalt. Then I need to locate my correct input.conf and put an stanza similar to [monitor::/path/to/folder/*.csv]
crcSalt=

So locate the correct input.conf, you can use the command $ find . -name inputs.conf -print...