topic Re: CSV files limitation in Splunk Search

CSV files limitation

jiaqya — Mon, 23 Apr 2018 11:32:49 GMT

Is there a limit associated with csv file for indexing on forwarder.
i find many of the csv files are skipping indexing randomly..

any suggestions on what could cause it.
in a previous question i did post the cpu/ram was below the minimum requirement for the forwarder.
could this be a potential reason....

Re: CSV files limitation

p_gurav — Mon, 23 Apr 2018 11:53:16 GMT

Is there any error in _internal index?

Re: CSV files limitation

ssadanala1 — Mon, 23 Apr 2018 13:22:21 GMT

Hi ,
The reason might be not adding crcSalt or crcinitlength in inputs.conf .

crcSalt =
* Use this setting to force the input to consume files that have matching CRCs
(cyclic redundancy checks).
* (The input only performs CRC checks against, by default, the first 256
bytes of a file. This behavior prevents the input from indexing the same
file twice, even though you may have renamed it -- as, for example, with
rolling log files. However, because the CRC is based on only the first
few lines of the file, it is possible for legitimately different files
to have matching CRCs, particularly if they have identical headers.)

Can you please add crcSalt = .

If the trick doesn't work please share how your inputs.conf looks like ?

Thanks

Re: CSV files limitation

jiaqya — Wed, 25 Apr 2018 10:09:05 GMT

crcSALT didnt help either.
for some reason after the service restart splunk had stopped indexing the csv files.
however strange, after 3-4 days everything is back to normal.
have no idea what caused it and what was the fix. however going ahead with upgrading the cpu/ram for the forwarders...