topic Re: CSV Header Issues in Splunk Enterprise

Any advice on how to resolve multiple CSV header issues?

andrew_burnett — Fri, 14 Apr 2023 20:49:10 GMT

We are getting multiple errors like this

Corrupt csv header in CSV file , 2 columns with the same name

However we have so many CSV files that finding them will be all but impossible.

Can someone provide advice on how to find them?

Re: CSV Header Issues

woodcock — Fri, 14 Apr 2023 19:46:23 GMT

Assuming that your OS is unix/linux, assuming that your CSV files use standard filenaming conventions (i.e. *.csv), assuming that your CSV files are standard with a header on the first line, assuming that the source files still exist, you can use the following CLI commands to identify problematic files:

find ${SPLUNK_HOME}/etc/apps/*/lookups -name *.csv -exec head -1 {} \; | tr ',' '\n' | sort| uniq -d

This will tell you the duplicated field, e.g. "foo". Then take that and do this to find the file (or a small pile to peek through):

for FILE in $(find ${SPLUNK_HOME}/lookups -name *.csv -exec grep -il foo {} \;); do echo ${FILE}; head -1 ${FILE} | tr ',' '\n' | sort | uniq -d; done

Here are some other tips:

View solution in original post

Re: CSV Header Issues

andrew_burnett — Fri, 14 Apr 2023 19:29:54 GMT

So the first one command, every word it brings back is a duplicated one?

Re: CSV Header Issues

woodcock — Fri, 14 Apr 2023 19:31:36 GMT

Exactly.

Re: CSV Header Issues

andrew_burnett — Fri, 14 Apr 2023 19:35:36 GMT

Well see we are trying to find specific keywords, so I know like one I'm trying to test. When I run your second command, it pulls in a ton of CSV files. Checking one, and the word isn't in the CSV header at all?

Re: CSV Header Issues

andrew_burnett — Fri, 14 Apr 2023 19:36:42 GMT

Oh I see it now, the word is in the CSV file itself. But I'm only concerned with the headers, is that not what the alert means?

Re: CSV Header Issues

woodcock — Fri, 14 Apr 2023 19:47:02 GMT

Yes. I updated my answer to help better.