topic Re: Columns missing while using the Add Data to upload a csv file in Getting Data In

Columns missing while using the Add Data to upload a csv file

jerrin — Fri, 13 Dec 2019 22:11:14 GMT

I have uploaded a csv file through the Splunk Web which has 157 columns and 4000 events.
I noticed that while directing to the index only 105 columns are visible with 4000 events.
Is there any limitation in splunk? And this is not the first time too I noticed this. Kindly help.

Re: Columns missing while using the Add Data to upload a csv file

woodcock — Sun, 15 Dec 2019 06:57:49 GMT

Almost always CSV files should be uploaded as lookup files, not event data. I am not sure what happened when you indexed this file, but try using it as lookup file instead and they try |inputlookup YourLookupFileName.csv and see if it works better (it probably will).

Re: Columns missing while using the Add Data to upload a csv file

jerrin — Mon, 16 Dec 2019 17:46:10 GMT

Thanks @woodcock. But this do not give a clarity why the columns are truncated.
I ingest csv file from the AWS S3 into splunk always.
These feeds do not qualify to be a Lookup.
Please help to understand why the truncation of columns.

Re: Columns missing while using the Add Data to upload a csv file

to4kawa — Mon, 16 Dec 2019 19:23:45 GMT

|inputlookup your_AWS_S3.csv
|head 1
|transpose

and

 index=your_index
|head 1
|transpose

When compared, there may be characters that cannot be used in field names.

Re: Columns missing while using the Add Data to upload a csv file

pgoldweic — Wed, 30 Jun 2021 20:19:47 GMT

I seem to be experiencing the same issue as reported here with a very small/simple csv file that suggests to me that only csv columns which have *at least* one non-empty value in them are being imported. Could somebody please confirm whether this is expected behavior (hopefully not) or some sort of bug?.