Security

Splunk 6.3 Just loaded S.o.S so why am getting "empty splunk_forwarders_cache.csv" file warnings?

Builder

I just loaded a brand new instance of Splunk Enterprise 6.3. I also loaded the Splunk S.o.S 3.2.1 app. While monitoring the splunkd.log file I noticed that I'm getting a ton of "empty splunk_forwarders_cache.csv" warnings.

    10-09-2015 14:04:08.848 -0400 WARN  SearchResults - /opt/app/splunk/etc/apps/sos/lookups/splunk_forwarders_cache.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
    10-09-2015 14:04:09.848 -0400 WARN  SearchResults - /opt/app/splunk/etc/apps/sos/lookups/splunk_forwarders_cache.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
    10-09-2015 14:04:10.848 -0400 WARN  SearchResults - /opt/app/splunk/etc/apps/sos/lookups/splunk_forwarders_cache.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header

Now, these warnings only showed up when I tried to turn on SSL with the entry below in "./etc/system/local/web.conf"

[settings]
enableSplunkWebSSL = 1

But when I reset enableSplunkWebSSL back to 0 and restarted Splunk, the warnings still kept on coming.

Now I am having issues with SSL and my browser, but I didn't think there should be problems when I disabled that.

Any ideas?

0 Karma
1 Solution

SplunkTrust
SplunkTrust

Hi OldManEd, my expectation is that S.o.S. runs a job at some point to populate that lookup with forwarders it has found. It might be that this is a single instance without any forwarders, and so it has nothing to populate that lookup with.

If you do have forwarders sending in data, you could start to trace the config for any other mentions of this lookup. Go to $SPLUNKHOME/etc and run:

grep -i splunk_forwarders_cache ./*/*/*

for instance, or similar strings such as that to find other clues.

View solution in original post

Builder

Now I'm getting the same thing when I tried to use the Management Console. The warning shows up every second and I can't turn it off:

10-12-2015 16:49:26.002 -0400 WARN  SearchResults - /opt/app/splunk/etc/apps/splunk_management_console/lookups/dmc_forwarder_assets.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
10-12-2015 16:49:27.002 -0400 WARN  SearchResults - /opt/app/splunk/etc/apps/splunk_management_console/lookups/dmc_forwarder_assets.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
10-12-2015 16:49:28.002 -0400 WARN  SearchResults - /opt/app/splunk/etc/apps/splunk_management_console/lookups/dmc_forwarder_assets.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
10-12-2015 16:49:29.002 -0400 WARN  SearchResults - /opt/app/splunk/etc/apps/splunk_management_console/lookups/dmc_forwarder_assets.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
10-12-2015 16:49:30.003 -0400 WARN  SearchResults - /opt/app/splunk/etc/apps/splunk_management_console/lookups/dmc_forwarder_assets.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
10-12-2015 16:49:31.002 -0400 WARN  SearchResults - /opt/app/splunk/etc/apps/splunk_management_console/lookups/dmc_forwarder_assets.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
10-12-2015 16:49:32.001 -0400 WARN  SearchResults - /opt/app/splunk/etc/apps/splunk_management_console/lookups/dmc_forwarder_assets.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
10-12-2015 16:49:33.002 -0400 WARN  SearchResults - /opt/app/splunk/etc/apps/splunk_management_console/lookups/dmc_forwarder_assets.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
10-12-2015 16:49:34.002 -0400 WARN  SearchResults - /opt/app/splunk/etc/apps/splunk_management_console/lookups/dmc_forwarder_assets.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
10-12-2015 16:49:34.998 -0400 WARN  SearchResults - /opt/app/splunk/etc/apps/splunk_management_console/lookups/dmc_forwarder_assets.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
0 Karma

SplunkTrust
SplunkTrust

Hi OldManEd, my expectation is that S.o.S. runs a job at some point to populate that lookup with forwarders it has found. It might be that this is a single instance without any forwarders, and so it has nothing to populate that lookup with.

If you do have forwarders sending in data, you could start to trace the config for any other mentions of this lookup. Go to $SPLUNKHOME/etc and run:

grep -i splunk_forwarders_cache ./*/*/*

for instance, or similar strings such as that to find other clues.

View solution in original post

Builder

Muebel,
You are correct. I'm just in the process of setting up this instance so I have no forwarders configured yet. When I looked at the "splunk_forwarders_cache.csv", it was there, but completely empty. I would think that even if this was the case, if the file is created, it would, at least, have the column "headers" by default.

Does anyone know what these column headers are for s.o.s? I was thinking that I could create a blank csv file, with the headers, to eliminate this repeating warning.

0 Karma

Splunk Employee
Splunk Employee

That particular lookup serves as an asset table for forwarder instances that are known to the S.o.S app. The columns it contains are described in $SPLUNK_HOME/etc/apps/sos/lookups/splunk_servers_cache.csv.spec, which shares the same columns.

The main reason why we cannot ship that file by default with the S.o.S app is simply that it would overwrite what your instance has already generated, as there is no concept of "default" and "local" spaces for lookup files. By all means, do add a header line to the lookup file to make the message go away.

0 Karma

Builder

Hexx,

Hey thanks for the explanation. But your answer creates 3 more questions from me:

1, If I manually create the CSV file with the headers, will whatever process that created it in the first place overwrite it again? Or was the process that creates it a one-time run? I can't imagine it's a one-time run because the the follow-up question would be what about updates? What happens when forwarders are added or deleted? I still don't have any forwarders established yet so I know that's going to change the status of this issue.

  1. If I add nothing to the file manually, will the warnings go away when I finally do add some forwarders? Will the issue clear itself?

  2. Now I noticed that when I shut down S.o.S and started to work with Management Console, I saw similar warnings. Are these two applications related somehow? It appears that the Splunk Management Console offers similar data to S.o.S. Is MC a replacement app?

~Ed

0 Karma

Splunk Employee
Splunk Employee
  • 1: That particular lookup file is maintained by a scheduled search named "sos_refresh_splunk_forwarders_cache" and it is updated every 15 minutes by default.
  • 1bis: I would expect the warnings to go away as soon as you have a lookup file with just the header, or one with forwarder records as well.
  • 2: Good eye! S.o.S is the ancestor of the Distributed Management Console and some people who developed the former (such as yours truly) have participated in the engineering of the latter. In that sense, you will find them occasionally related. And most definitely, the DMC is poised to replace S.o.S as it delivers most of the same content and much, much more!
0 Karma

Builder

Hexx,
Thanks for the quick reply. This explains everything I questions about. I think I'll just uninstall S.o.S an use Distributed Management Console, and hope I get access to some of the remote servers I need to add the Splunk forwarder to soon.
~Ed

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!