CSV header extraction limit?

[ebailey]

I am trying to on-board a new data source to Splunk. It is a CSV file with 350 headers records. I setup an inputs and a props for the file.

props.conf

[test]
DATETIME_CONFIG = CURRENT
INDEXED_EXTRACTIONS = csv
KV_MODE = none
NO_BINARY_CHECK = 1
SHOULD_LINEMERGE = false
pulldown_type = 1
TRUNCATE = 10000000

The extraction of the headers into fields works partially. I only get the first 30 fields/headers. How can I get all the headers to extract?

Any ideas?

Thanks!

[bhawkins1]

See http://docs.splunk.com/Documentation/Splunk/6.5.2/Data/Extractfieldsfromfileswithstructureddata#Stru... for the right answer.

Specificaly: props.conf wants to have [kv]'s limit = <max_cols> set to something big. We're using 1000.

[kendrickt]

Have you tried passing them to a transforms.conf and listing the fields?

It will be a pain, but you'll only have to do it once and it should work!

props.conf

[test]
all your stuff
TRANSFORMS-new = testtransforms

tranforms.conf

[testtransforms]
DELIMS = ","
FIELDS = "field1", "field2", "field3" etc...

Might wanna cut/paste!

[ebailey]

That is what I was guessing I needed to do. Too bad the default csv sourcetype does not work. Thanks

[ebailey]

I made the change

props

[test]
DATETIME_CONFIG = CURRENT
INDEXED_EXTRACTIONS = csv
KV_MODE = none
NO_BINARY_CHECK = 1
SHOULD_LINEMERGE = false
pulldown_type = 1
TRUNCATE = 10000000
TRANSFORMS-new = testtransforms

transforms

[testtransforms]
REGEX = .
DELIMS = ","
FIELDS = "long list of values"

I am not getting the expected results. I only see the fields I saw before I manually defined the layout. Is the format correct?

[kendrickt]

I don't know what impact the REGEX field will have, but it shouldn't be required.

Here's a copy of a transforms.conf I use everyday

[xxx]
DELIMS=","
FIELDS="xxx1","xxx2","xxx3"

I would probably set TRUNCATE=0 too, but only if you're sure there'll be no garbage in the file.