Getting Data In

CSV header extraction limit?

ebailey
Communicator

I am trying to on-board a new data source to Splunk. It is a CSV file with 350 headers records. I setup an inputs and a props for the file.

props.conf

[test]
DATETIME_CONFIG = CURRENT
INDEXED_EXTRACTIONS = csv
KV_MODE = none
NO_BINARY_CHECK = 1
SHOULD_LINEMERGE = false
pulldown_type = 1
TRUNCATE = 10000000

The extraction of the headers into fields works partially. I only get the first 30 fields/headers. How can I get all the headers to extract?

Any ideas?

Thanks!

Tags (3)

bhawkins1
Communicator

See http://docs.splunk.com/Documentation/Splunk/6.5.2/Data/Extractfieldsfromfileswithstructureddata#Stru... for the right answer.

Specificaly: props.conf wants to have [kv]'s limit = <max_cols> set to something big. We're using 1000.

kendrickt
Path Finder

Have you tried passing them to a transforms.conf and listing the fields?

It will be a pain, but you'll only have to do it once and it should work!

props.conf

[test]
all your stuff
TRANSFORMS-new = testtransforms

tranforms.conf

[testtransforms]
DELIMS = ","
FIELDS = "field1", "field2", "field3" etc...

Might wanna cut/paste!

ebailey
Communicator

That is what I was guessing I needed to do. Too bad the default csv sourcetype does not work. Thanks

0 Karma

ebailey
Communicator

I made the change

props

[test]
DATETIME_CONFIG = CURRENT
INDEXED_EXTRACTIONS = csv
KV_MODE = none
NO_BINARY_CHECK = 1
SHOULD_LINEMERGE = false
pulldown_type = 1
TRUNCATE = 10000000
TRANSFORMS-new = testtransforms

transforms

[testtransforms]
REGEX = .
DELIMS = ","
FIELDS = "long list of values"

I am not getting the expected results. I only see the fields I saw before I manually defined the layout. Is the format correct?

0 Karma

kendrickt
Path Finder

I don't know what impact the REGEX field will have, but it shouldn't be required.

Here's a copy of a transforms.conf I use everyday

[xxx]
DELIMS=","
FIELDS="xxx1","xxx2","xxx3"

I would probably set TRUNCATE=0 too, but only if you're sure there'll be no garbage in the file.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...