Splunk Monitor File and Folder: Not uploading all files

[Chandras11]

HI All,

I am trying to monitor 3 CSVs from a same folder via Splunk : Settings -> Data Input -> Files & Directories.
My Files are Alpha1.csv, Beta1.csv and Gamma1.csv. and the sourcetpyes are Alpha , Beta and Gamma with a proper timestamp (predefined).
so in the Settings -> Data Input -> Files & Directories: I added the files as:-

Full path: C:\FileLOG\Alpha*.csv 
Whitelist:  Alpha
SourceType: Alpha
Index = Index1

Full path: C:\FileLOG\Beta*.csv 
Whitelist:  Beta
SourceType: Beta
Index = Index1

Full path: C:\FileLOG\Gamma*.csv 
Whitelist:  Gamma
SourceType: Gamma
Index = Index1

However, when I check Splunk, it shows me events only from C:\FileLOG\Gamma*.csv. Normal Upload of data is working fine and there is no issue with source type or index. Just the monitor function won't upload all files, although different rules are applied.

thanks for the help and my apologies for such a long post.

[Chandras11]

So, if anyone gets the same issue, the workaround is:-
First run the query in splunk search head:-
index=_internal log_level=warn* OR log_level=error and check for the time, when you put the CSVs in the folder.
You will get the possible error or warning. I got the following:
06-14-2018 10:43:10.079 +0200 ERROR TailReader - File will not be read, seekptr checksum did not match (file=C:\FileLOG\Beta1.csv ). Last time we saw this initcrc, filename was different. You may wish to use larger initCrcLen for this sourcetype, or a CRC salt on this source. Consult the documentation or file a support case online at http://www.splunk.com/page/submit_issue for more info.

So I had the issue with crcSalt. Then I need to locate my correct input.conf and put an stanza similar to [monitor::/path/to/folder/*.csv]
crcSalt=

So locate the correct input.conf, you can use the command $ find . -name inputs.conf -print...

[jkat54 · Accepted solution]

check the results of this search for crcinit issues, permissions isssues, etc.

index=_internal log_level=warn* OR log_level=error

[Chandras11]

Thanks for it. I found the error as:

06-14-2018 10:43:10.159 +0200 ERROR TailReader - File will not be read, is too small to match seekptr checksum (file=C:\FileLOG\Alpha1.csv). Last time we saw this initcrc, filename was different. You may wish to use larger initCrcLen for this sourcetype, or a CRC salt on this source. Consult the documentation or file a support case online at http://www.splunk.com/page/submit_issue for more info.

06-14-2018 10:43:10.079 +0200 ERROR TailReader - File will not be read, seekptr checksum did not match (file=C:\FileLOG\Beta1.csv ). Last time we saw this initcrc, filename was different. You may wish to use larger initCrcLen for this sourcetype, or a CRC salt on this source. Consult the documentation or file a support case online at http://www.splunk.com/page/submit_issue for more info.

[FrankVl]

Sounds like the first bits of those CSV files may be too similar. Increasing the initCrcLen or adding crcSalt = <SOURCE> (literally like that) in the inputs.conf for each of those inputs might resolve your issue.

[Chandras11]

Thanks a lot for the comment. Now my input.conf looks like:

[default]
host = CPX-XXXXXXXX
crcSalt =

Do I need to specify all files and add crcSalt for all of them? Sorry but the Splunk answer is removing the SOURCE keyword automatically 🙂

[FrankVl]

No, you need to find the inputs.conf file where the inputs are already configured (as mentioned in my other comment, my guess would be they are under etc/apps/search/local/). And then add the crcSalt setting there in each section for the three csv inputs.

[Chandras11]

Sorry but there is no input.conf under etc/apps/search/local/. I found one under Splunk\etc\system\local

but there is no different lines for different sourcetype. I believe I am not looking at the correct place. 😞

[FrankVl]

Have a look on the data inputs page that lists all the configured inputs. That should also mention in which app the config was stored. That should help you find it on the filesystem.

[Chandras11]

Thanks a lot Frank. I got it with the command: $ find . -name inputs.conf -print . It was udner the C:\Program Files\Splunk\etc\apps\MYAPP\local folder

[jkat54]

So now you will have a stanza like this:

 [monitor::/path/to/folder/*.log]
 crcSalt=<SOURCE>

And that will fix the problem after restarting.

You may have to clear the fish bucket or use another index name to get it to reload the data.

[FrankVl]

Can you share a screenshot of the relevant part of the Settings -> Data Input -> Files & Directories page, showing the entries for those 3 inputs? (or if you know how to find it: share the inputs.conf file that was generated for this).

[Chandras11]

I cannot share the screenshot from Splunk (Sensitive Data), but I can share entries in input.conf in

C:\Program Files\Splunk\etc\system\local
[default]
host = CPX-XXXXXXXX

[FrankVl]

That's not the relevant inputs.conf file. Not sure in which app the config was added, but you might want to take a look in etc/apps/search/local/inputs.conf