Warning in internal log: unable to parse site_label

[mortf]

I recently noticed a huge amount of warnings in the _internal logs for our search heads. events are all like this:

02-04-2021 12:22:08.485 +0300 WARN  SearchResultsFiles - Unable to parse site_label, label=invalid due to err="Invalid site id: invalid"

We are running a distributed environment with a search head cluster and all installations are Splunk 8.1.1. The warnings are logged only on the search heads.

When investigating i see this has occured for quite som time but i'm very qurious as to what this means.  There are no other indications in the _internal log that hints to why this warning keep appearing. I have however discovered that it seems to maybe be related to lookups and perhaps the kvstore. The reason i think so is that i can't force this warning when doing normal searches, but when i open dashbords that uses searches with macros and lookups they appear immediately. I've tried several different dashboards and searches and it seems consistent that anything with a lookup will produce this warning. 

I'm further thinking this might have happened when we upgraded to Splunk 8.1.1 recently. I've got two standalone servers for test purposes where one is running Splunk 8.1.1 and the other one is running Splunk 8.1.0.1
I have not been able to force this warning on the Splunk instance running 8.1.0.1 as of yet, but the one running 8.1.1 will have these warnings when i open dashbords and advanced searches.

I have not found anything in the Splunk "known issues" about this warning specifically. I don't even know if it causes any problems other than filling up the _internal log (There are noe issues with our environment relating to this warning as far as i know).

So i was wondering if anyone else have been experiencing these warnings, know what they are and know how to stop them? In peak search time there can be several million events per hour. 

One thing i have not yet tried, but will try as soon as possible, is to upgrade one of the standalone servers to Splunk 8.1.2 and see if that fixes things.

[sansay1]

This appears to still not have been fixed, even though we are at version 8.2.7.
This is very disappointing from Splunk as it fills our logs with garbage and important logs get pushed out of the historical records. 
Splunk engineers: This needs to be fixed now.

[dfronck]

V8.2.6 - Still not fixed.

https://docs.splunk.com/Documentation/Splunk/8.2.6/ReleaseNotes/Knownissues 

SPL-212495, SPL-196040, SPL-219811
Excessive logging 'WARN SearchResultsFiles Unable to parse site_label, label=invalid due to err="Invalid site id: invalid"' for SearchResultsFiles
Workaround: none

[vgrote]

Plus ca change ...

https://docs.splunk.com/Documentation/Splunk/9.0.0/ReleaseNotes/Knownissues

2021-09-22

SPL-212495, SPL-196040, SPL-219811

Excessive logging 'WARN SearchResultsFiles Unable to parse site_label, label=invalid due to err="Invalid site id: invalid"' for SearchResultsFiles Workaround: none

[richgalloway]

Check the site settings in server.conf on all search heads.

[mortf · Accepted solution]

The server.conf i /etc/system/local is the same on all search heads and there are no specific site settings there, so all of them should follow the default site settings from the /etc/system/default server.conf

I asked this same question in slack and someone there told me that the issue i'm experiencing is a bug in the current major release version of Splunk. The issue will be fixed in the next major release version and so i'm choosing this as the answer to my question. There does not seem to be a viable workaround.

[orion44]

@mortf I'm also experiencing this issue, the following error message is flooding my splunkd.log:

WARN SearchResultsFiles - Unable to parse site_label, label=invalid due to err="Invalid site id: invalid"

Can you confirm upgrading to a later version of Splunk resolved your issue? If so, what version did you upgrade to?

[mortf]

Hi.

Upgrading to a new version of Splunk did not resolve this issue. The newest splunk version has this listet under known issues as well. Check out SPL-212495 and SPL-196040.