Splunk Enterprise

WARN SearchResultsCSVSerializer - CSV file contains invalid field '', ignoring column.

vgrote
Engager

Hi,

we are seeing > 70,000 of these messages per day per instance on several Searchheads on Splunk 8.0.5.1 and SUSE Linux 12:

WARN SearchResultsCSVSerializer - CSV file  contains invalid field '', ignoring column.

(there are actually two spaces after "file", and '' are two single quotes)

In a Searchhead Cluster only the captain seems to report this. If I clone the Splunk installation from an affected Searchhead to another, similar but unaffected one that doesn't show the symptoms, I cannot reproduce the messages there.

At startup it kicks in around here:

04-12-2021 16:56:47.361 +0200 INFO ServerRoles - Declared role=search_head.
04-12-2021 16:56:49.680 +0200 INFO ServerRoles - Declared role=kv_store.
04-12-2021 16:56:49.684 +0200 INFO CertStorageProvider - Updating status from starting to ready
04-12-2021 16:56:49.684 +0200 INFO CertStorageProvider - Updating status from starting to ready
04-12-2021 16:56:49.684 +0200 INFO Rsa2FA - Could not find [externalTwoFactorAuthSettings] in authentication stanza.
04-12-2021 16:56:50.911 +0200 WARN SearchResultsCSVSerializer - CSV file  contains invalid field '', ignoring column.

When the system goes down it stops here:

04-12-2021 16:56:02.889 +0200 WARN SearchResultsCSVSerializer - CSV file  contains invalid field '', ignoring column.
04-12-2021 16:56:03.831 +0200 INFO loader - Shutdown HTTPDispatchThread
04-12-2021 16:56:03.831 +0200 INFO ShutdownHandler - Shutting down splunkd

a) Has anyone seen this too? And if so, fixed it? How?

b) How can I get Splunk to report some more detail, like who wants to open that file? I found no useful information on the SearchResultsCSVSerializer and "strace" on Linux did not provide any clue for me either.

Thanks in advance

Volkmar

0 Karma
1 Solution

garias_splunk
Splunk Employee
Splunk Employee

The issue ended up being an extra comma at the end of the first row of the CSV file. Splunk understands there is an extra column after it. 

columnheader1,columnheader2,columnheader3,columnheader4,      <-- that last comma causes the issue

value11,value12,value13,value14

value21,value22,value23,value24

...

 

 

 

View solution in original post

garias_splunk
Splunk Employee
Splunk Employee


The process SearchResultsCSVSerializer is used to to read csv from disk. What this message is saying is there is something wrong with one field in a CSV file. The problem could be something like this:
- A column within a CSV file where there is not an associated column name above (heather),
- A comma in the data that's not surrounded by double quotes, which leads to what Splunk thinks is a field with no column name.

Your message is not showing the process that is calling it. In normal circumstances it should be something like this:
04-12-2021 16:56:50.911 +0200 WARN SearchResultsCSVSerializer [35665 MongoModificationsTrackerThread] - CSV file contains invalid field '', ignoring column.

That message is logged both in splunkd.log and search.log, in your description the log extraction comes from splunkd.log. Looking for the same error in search.log can give you more information about the failing lookup:


04-18-2021 19:28:35.836 INFO CsvDataProvider - Reading schema for lookup table='windows_eventcode', file size=7135, modtime=1607355381
04-18-2021 19:28:35.836 WARN SearchResultsCSVSerializer - CSV file contains invalid field '', ignoring column.

Probably your issue is within windows_eventcode.csv. Fix that file, or rebuilt it, or delete it as per your convenience.

If that is not the CSV you are looking for, you can manually look for it in your disk. In order to get rid of this issue, you need to identify which CSV file causes this problem. You can use a 'grep' or 'find' command for that as reported here:
https://community.splunk.com/t5/Splunk-Search/quot-Corrupt-csv-header-quot-how-to-find-the-corrupted...

Also, there is an app that is used to manage the lookup files and can help to identify the conflictive lookup file. This app is not Supported by Splunk but can be used:
https://splunkbase.splunk.com/app/1724/

0 Karma

garias_splunk
Splunk Employee
Splunk Employee

The issue ended up being an extra comma at the end of the first row of the CSV file. Splunk understands there is an extra column after it. 

columnheader1,columnheader2,columnheader3,columnheader4,      <-- that last comma causes the issue

value11,value12,value13,value14

value21,value22,value23,value24

...

 

 

 

View solution in original post

Take the 2021 Splunk Career Survey

Help us learn about how Splunk has
impacted your career by taking the 2021 Splunk Career Survey.

Earn $50 in Amazon cash!