topic Re: Ignoring header in the csv file in Getting Data In

Ignoring header in the csv file

kiril123 — Tue, 29 Sep 2020 14:18:06 GMT

I want to index and search csv files in splunk.

Each file has a header at the first line:

number1,number2, number3,
1,2,3
4,5,6

I've created a custom csv sourcetype in props.conf and defined custom fields i want to use instead of the header in transforms.conf:

props.conf:

[custom-csv]
DATETIME_CONFIG = CURRENT
INDEXED_EXTRACTIONS = csv
NO_BINARY_CHECK = true
category = Structured
pulldown_type = true
SHOULD_LINEMERGE = false
disabled = false
KV_MODE = none
REPORT-custom= REPORT-custom

transforms.conf:

[REPORT-custom]
DELIMS = ","
FIELDS = number1, number2, number3

However when i run a search i can still see the original field names extracted from the header plus the new ones i have defined in transforms.conf.

Is there a way to make Splunk to ignore the header line?

Re: Ignoring header in the csv file

richgalloway — Thu, 01 Jun 2017 17:46:16 GMT

Try using the PREAMBLE_REGEX attribute in props.conf. Something like this. You may need to adjust the regex.

[custom-csv]
DATETIME_CONFIG = CURRENT
INDEXED_EXTRACTIONS = csv
PREAMBLE_REGEX = number1, number2, number3
NO_BINARY_CHECK = true
category = Structured
pulldown_type = true
SHOULD_LINEMERGE = false
disabled = false
KV_MODE = none
REPORT-custom= REPORT-custom

Re: Ignoring header in the csv file

kiril123 — Thu, 01 Jun 2017 18:08:30 GMT

I have tried using PREAMBLE_REGEX but now Splunk uses the next line as a header so i get 1,2,3 as the search fields, rather than values.

Re: Ignoring header in the csv file

richgalloway — Fri, 02 Jun 2017 12:24:57 GMT

From your examples, it appears you are specifying the same field names that are already in the CSV. If so, don't bother. Omit the transform and let Splunk process the file normally.

Re: Ignoring header in the csv file

michael_sleep — Tue, 29 Sep 2020 14:18:29 GMT

I think the problem is that you don't need to use INDEXED_EXTRACTIONS at all if you don't intend to have Splunk extract the data based on the field headers at the top of the CSV file. I think DELIMS is also a search-time extraction and it doesn't have any reliance on INDEXED_EXTRACTIONS. So you can use PREAMBLE_REGEX but just don't use INDEXED_EXTRACTIONS. Don't treat it like a CSV file, just treat it like a log file.