(there are actually two spaces after "file", and '' are two single quotes)
In a Searchhead Cluster only the captain seems to report this. If I clone the Splunk installation from an affected Searchhead to another, similar but unaffected one that doesn't show the symptoms, I cannot reproduce the messages there.
At startup it kicks in around here:
04-12-2021 16:56:47.361 +0200 INFO ServerRoles - Declared role=search_head. 04-12-2021 16:56:49.680 +0200 INFO ServerRoles - Declared role=kv_store. 04-12-2021 16:56:49.684 +0200 INFO CertStorageProvider - Updating status from starting to ready 04-12-2021 16:56:49.684 +0200 INFO CertStorageProvider - Updating status from starting to ready 04-12-2021 16:56:49.684 +0200 INFO Rsa2FA - Could not find [externalTwoFactorAuthSettings] in authentication stanza. 04-12-2021 16:56:50.911 +0200 WARN SearchResultsCSVSerializer - CSV file contains invalid field '', ignoring column.
When the system goes down it stops here:
04-12-2021 16:56:02.889 +0200 WARN SearchResultsCSVSerializer - CSV file contains invalid field '', ignoring column. 04-12-2021 16:56:03.831 +0200 INFO loader - Shutdown HTTPDispatchThread 04-12-2021 16:56:03.831 +0200 INFO ShutdownHandler - Shutting down splunkd
a) Has anyone seen this too? And if so, fixed it? How?
b) How can I get Splunk to report some more detail, like who wants to open that file? I found no useful information on the SearchResultsCSVSerializer and "strace" on Linux did not provide any clue for me either.
The process SearchResultsCSVSerializer is used to to read csv from disk. What this message is saying is there is something wrong with one field in a CSV file. The problem could be something like this: - A column within a CSV file where there is not an associated column name above (heather), - A comma in the data that's not surrounded by double quotes, which leads to what Splunk thinks is a field with no column name.
Your message is not showing the process that is calling it. In normal circumstances it should be something like this: 04-12-2021 16:56:50.911 +0200 WARN SearchResultsCSVSerializer [35665 MongoModificationsTrackerThread] - CSV file contains invalid field '', ignoring column.
That message is logged both in splunkd.log and search.log, in your description the log extraction comes from splunkd.log. Looking for the same error in search.log can give you more information about the failing lookup:
04-18-2021 19:28:35.836 INFO CsvDataProvider - Reading schema for lookup table='windows_eventcode', file size=7135, modtime=1607355381 04-18-2021 19:28:35.836 WARN SearchResultsCSVSerializer - CSV file contains invalid field '', ignoring column.
Probably your issue is within windows_eventcode.csv. Fix that file, or rebuilt it, or delete it as per your convenience.
Also, there is an app that is used to manage the lookup files and can help to identify the conflictive lookup file. This app is not Supported by Splunk but can be used: https://splunkbase.splunk.com/app/1724/