topic Re: I cannot get splunk to ingest my csv files! in Getting Data In

I cannot get splunk to ingest my csv files!

jedatt01 — Tue, 23 May 2017 20:25:30 GMT

I have some csv files that have 30+ columns and I cannot get splunk to ingest them. I keep getting crc errors. I've tried to use crcSalt and initCrcLength but I keep getting the same error message below in splunkd.log. The files can be very similar sometimes. Anyone have any ideas?

splunkd.log

05-23-2017 16:15:43.313 -0400 ERROR TailReader - File will not be read, seekptr checksum did not match (file=*****\Public\Test\test.csv).  Last time we saw this initcrc, filename was different.  You may wish to use larger initCrcLen for this sourcetype, or a CRC salt on this source.  Consult the documentation or file a support case online at http://www.splunk.com/page/submit_issue for more info.

inputs.conf stanza

[batch://\****\Public\Test\*.csv]
crcSalt = <SOURCE>
move_policy = sinkhole
sourcetype = pub_audit
index = main
DATETIME_CONFIG = CURRENT

Re: I cannot get splunk to ingest my csv files!

somesoni2 — Tue, 23 May 2017 21:27:48 GMT

Give this a try

 [batch://\****\Public\Test\*.csv]
 crcSalt = <SOURCE>
 initCrcLength = 4999
 move_policy = sinkhole
 sourcetype = pub_audit
 index = main

The DATETIME_CONFIG attribute is a props.conf property, not inputs.conf.

Re: I cannot get splunk to ingest my csv files!

woodcock — Tue, 23 May 2017 22:03:16 GMT

Do as @somesoni2 says and clean up the garbage, deploy this inputs.conf file to your forwarder, restart all splunk instances on that forwarder, make sure that the files are disappearing (if not, then they are not being forwarded). MAKE SURE that each file has a different name that is never, ever, ever, ever recycled (that is what the crcSalt = <SOURCE> line does).

Also, your quadruple-asterisks is quite strange; why is it that way? Are you missing backslash characters between them? Are you trying to match a file that literally has a string of asterisks for a directory name? Are you trying to match directories that are 4-characters long?

Re: I cannot get splunk to ingest my csv files!

jedatt01 — Tue, 29 Sep 2020 14:11:40 GMT

The only way I was able to get this to work was to use the CHECK_METHOD = entire_md5 in the props.conf. That is ok because my files are always small.