topic Re: CSV header extraction limit? in Getting Data In

CSV header extraction limit?

ebailey — Mon, 28 Sep 2020 18:48:41 GMT

I am trying to on-board a new data source to Splunk. It is a CSV file with 350 headers records. I setup an inputs and a props for the file.

props.conf

[test]
DATETIME_CONFIG = CURRENT
INDEXED_EXTRACTIONS = csv
KV_MODE = none
NO_BINARY_CHECK = 1
SHOULD_LINEMERGE = false
pulldown_type = 1
TRUNCATE = 10000000

The extraction of the headers into fields works partially. I only get the first 30 fields/headers. How can I get all the headers to extract?

Any ideas?

Thanks!

Re: CSV header extraction limit?

kendrickt — Fri, 30 Jan 2015 20:34:15 GMT

Have you tried passing them to a transforms.conf and listing the fields?

It will be a pain, but you'll only have to do it once and it should work!

props.conf

[test]
all your stuff
TRANSFORMS-new = testtransforms

tranforms.conf

[testtransforms]
DELIMS = ","
FIELDS = "field1", "field2", "field3" etc...

Might wanna cut/paste!

Re: CSV header extraction limit?

ebailey — Mon, 02 Feb 2015 15:37:57 GMT

That is what I was guessing I needed to do. Too bad the default csv sourcetype does not work. Thanks

Re: CSV header extraction limit?

ebailey — Mon, 28 Sep 2020 18:51:22 GMT

I made the change

props

[test]
DATETIME_CONFIG = CURRENT
INDEXED_EXTRACTIONS = csv
KV_MODE = none
NO_BINARY_CHECK = 1
SHOULD_LINEMERGE = false
pulldown_type = 1
TRUNCATE = 10000000
TRANSFORMS-new = testtransforms

transforms

[testtransforms]
REGEX = .
DELIMS = ","
FIELDS = "long list of values"

I am not getting the expected results. I only see the fields I saw before I manually defined the layout. Is the format correct?

Re: CSV header extraction limit?

kendrickt — Wed, 04 Feb 2015 16:46:50 GMT

I don't know what impact the REGEX field will have, but it shouldn't be required.

Here's a copy of a transforms.conf I use everyday

[xxx]
DELIMS=","
FIELDS="xxx1","xxx2","xxx3"

I would probably set TRUNCATE=0 too, but only if you're sure there'll be no garbage in the file.

Re: CSV header extraction limit?

bhawkins1 — Fri, 24 Feb 2017 10:20:12 GMT

See http://docs.splunk.com/Documentation/Splunk/6.5.2/Data/Extractfieldsfromfileswithstructureddata#Structured_data_files_with_large_numbers_of_columns_might_not_display_all_extracted_fields_in_Splunk_Search for the right answer.

Specificaly: props.conf wants to have [kv]'s limit = <max_cols> set to something big. We're using 1000.