Splunk Enterprise

Why are we seeing this message: WARN SearchResultsCSVSerializer - CSV file contains invalid field '', ignoring column.

vgrote
Path Finder

Hi,

we are seeing > 70,000 of these messages per day per instance on several Searchheads on Splunk 8.0.5.1 and SUSE Linux 12:

WARN SearchResultsCSVSerializer - CSV file  contains invalid field '', ignoring column.

(there are actually two spaces after "file", and '' are two single quotes)

In a Searchhead Cluster only the captain seems to report this. If I clone the Splunk installation from an affected Searchhead to another, similar but unaffected one that doesn't show the symptoms, I cannot reproduce the messages there.

At startup it kicks in around here:

04-12-2021 16:56:47.361 +0200 INFO ServerRoles - Declared role=search_head.
04-12-2021 16:56:49.680 +0200 INFO ServerRoles - Declared role=kv_store.
04-12-2021 16:56:49.684 +0200 INFO CertStorageProvider - Updating status from starting to ready
04-12-2021 16:56:49.684 +0200 INFO CertStorageProvider - Updating status from starting to ready
04-12-2021 16:56:49.684 +0200 INFO Rsa2FA - Could not find [externalTwoFactorAuthSettings] in authentication stanza.
04-12-2021 16:56:50.911 +0200 WARN SearchResultsCSVSerializer - CSV file  contains invalid field '', ignoring column.

When the system goes down it stops here:

04-12-2021 16:56:02.889 +0200 WARN SearchResultsCSVSerializer - CSV file  contains invalid field '', ignoring column.
04-12-2021 16:56:03.831 +0200 INFO loader - Shutdown HTTPDispatchThread
04-12-2021 16:56:03.831 +0200 INFO ShutdownHandler - Shutting down splunkd

a) Has anyone seen this too? And if so, fixed it? How?

b) How can I get Splunk to report some more detail, like who wants to open that file? I found no useful information on the SearchResultsCSVSerializer and "strace" on Linux did not provide any clue for me either.

Thanks in advance

Volkmar

0 Karma
1 Solution

garias_splunk
Splunk Employee
Splunk Employee

The issue ended up being an extra comma at the end of the first row of the CSV file. Splunk understands there is an extra column after it. 

columnheader1,columnheader2,columnheader3,columnheader4,      <-- that last comma causes the issue

value11,value12,value13,value14

value21,value22,value23,value24

...

 

 

 

View solution in original post

911
Engager

This is all that's in the log file <splunk>/var/log/splunk/splunkd.log 

 

05-06-2022 12:35:57.137 +0200 WARN SearchResultsCSVSerializer [5148 MongoModificationsTrackerThread] - CSV file contains invalid field '', ignoring column.
05-06-2022 12:35:58.135 +0200 WARN SearchResultsCSVSerializer [5148 MongoModificationsTrackerThread] - CSV file contains invalid field '', ignoring column.
05-06-2022 12:35:59.137 +0200 WARN SearchResultsCSVSerializer [5148 MongoModificationsTrackerThread] - CSV file contains invalid field '', ignoring column.
05-06-2022 12:36:00.137 +0200 WARN SearchResultsCSVSerializer [5148 MongoModificationsTrackerThread] - CSV file contains invalid field '', ignoring column.

0 Karma

vgrote
Path Finder

Did you check search.log? Any result?

Did you run 

find ./ -type f -name '*.csv' | xargs grep ",$"

What was the result? 

0 Karma

garias_splunk
Splunk Employee
Splunk Employee


The process SearchResultsCSVSerializer is used to to read csv from disk. What this message is saying is there is something wrong with one field in a CSV file. The problem could be something like this:
- A column within a CSV file where there is not an associated column name above (heather),
- A comma in the data that's not surrounded by double quotes, which leads to what Splunk thinks is a field with no column name.

Your message is not showing the process that is calling it. In normal circumstances it should be something like this:
04-12-2021 16:56:50.911 +0200 WARN SearchResultsCSVSerializer [35665 MongoModificationsTrackerThread] - CSV file contains invalid field '', ignoring column.

That message is logged both in splunkd.log and search.log, in your description the log extraction comes from splunkd.log. Looking for the same error in search.log can give you more information about the failing lookup:


04-18-2021 19:28:35.836 INFO CsvDataProvider - Reading schema for lookup table='windows_eventcode', file size=7135, modtime=1607355381
04-18-2021 19:28:35.836 WARN SearchResultsCSVSerializer - CSV file contains invalid field '', ignoring column.

Probably your issue is within windows_eventcode.csv. Fix that file, or rebuilt it, or delete it as per your convenience.

If that is not the CSV you are looking for, you can manually look for it in your disk. In order to get rid of this issue, you need to identify which CSV file causes this problem. You can use a 'grep' or 'find' command for that as reported here:
https://community.splunk.com/t5/Splunk-Search/quot-Corrupt-csv-header-quot-how-to-find-the-corrupted...

Also, there is an app that is used to manage the lookup files and can help to identify the conflictive lookup file. This app is not Supported by Splunk but can be used:
https://splunkbase.splunk.com/app/1724/

0 Karma

garias_splunk
Splunk Employee
Splunk Employee

The issue ended up being an extra comma at the end of the first row of the CSV file. Splunk understands there is an extra column after it. 

columnheader1,columnheader2,columnheader3,columnheader4,      <-- that last comma causes the issue

value11,value12,value13,value14

value21,value22,value23,value24

...

 

 

 

cmeisch
Path Finder

How did you know which csv file?

 

0 Karma

vgrote
Path Finder

As garias_splunk wrote: 

you can try something like this command to find a .csv file that contains a comma followed by an end of line sign

find ./ -type f -name '*.csv' | xargs grep ",$"

 

0 Karma

911
Engager

How did you know which .csv file to check  ?

0 Karma

garias_splunk
Splunk Employee
Splunk Employee

it was long ago and I do not remember. Probably the path is written somewhere in the logs. But you can try something like this command to find a .csv file that contains a comma followed by an end of line sign

 

find ./ -type f -name '*.csv' | xargs grep ",$"

0 Karma

jaspal95
Loves-to-Learn Everything

Do you know how to do it for windows?

0 Karma

vgrote
Path Finder

Hi jaspal95,

sorry, I have no idea other than bringing a Unix shell to Windows.

VGVG

 

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...