topic Re: Why splunk is not able to index large csv files, in Getting Data In

Why splunk is not able to index large csv files,

premforsplunk — Tue, 12 Dec 2017 21:19:24 GMT

I'm trying to index a 3.5 GB csv file, but splunk is not reading it. Any clues ?

Re: Why splunk is not able to index large csv files,

mayurr98 — Wed, 13 Dec 2017 07:15:49 GMT

what configuration you have done to index this csv?
If connectivity and all is there then
add following command in your splunkforwarder using cli
./splunk add monitor -index index_name -sourcetype csv

also on the indexer create the same index that you specified as index_name

Re: Why splunk is not able to index large csv files,

MousumiChowdhur — Wed, 20 Dec 2017 09:41:48 GMT

Hi!

You can add the below configuration parameters in your limits.conf file of your app.

[kv]
limit =
maxcols =

The default values for limit is 100 and maxcols is 512. You should try indexing your csv by increasing the default values.

Link for your reference.
https://docs.splunk.com/Documentation/Splunk/7.0.1/Admin/Limitsconf

Thanks.

Re: Why splunk is not able to index large csv files,

nickhills — Wed, 20 Dec 2017 09:53:42 GMT

I had to do this once for a huge CSV, but I also has to prune a few rows, and didn't need half the columns, (so not exactly the same as your requirement)

I actually opted to do this with a python scripted input, which allowed me to pre-process the file as it went in, and dumped to stdout as key=value pairs.
Once it was completed I disabled the input, but it meant I had the ability to run it again if ever needed.
I never did need it again, but even so, It was time well spent making sure the data was concise when it went in.