Splunk Search

Warning in internal log: unable to parse site_label

mortf
Explorer

I recently noticed a huge amount of warnings in the _internal logs for our search heads. events are all like this:

02-04-2021 12:22:08.485 +0300 WARN  SearchResultsFiles - Unable to parse site_label, label=invalid due to err="Invalid site id: invalid"

We are running a distributed environment with a search head cluster and all installations are Splunk 8.1.1. The warnings are logged only on the search heads.

When investigating i see this has occured for quite som time but i'm very qurious as to what this means.  There are no other indications in the _internal log that hints to why this warning keep appearing. I have however discovered that it seems to maybe be related to lookups and perhaps the kvstore. The reason i think so is that i can't force this warning when doing normal searches, but when i open dashbords that uses searches with macros and lookups they appear immediately. I've tried several different dashboards and searches and it seems consistent that anything with a lookup will produce this warning. 

I'm further thinking this might have happened when we upgraded to Splunk 8.1.1 recently. I've got two standalone servers for test purposes where one is running Splunk 8.1.1 and the other one is running Splunk 8.1.0.1
I have not been able to force this warning on the Splunk instance running 8.1.0.1 as of yet, but the one running 8.1.1 will have these warnings when i open dashbords and advanced searches.

I have not found anything in the Splunk "known issues" about this warning specifically. I don't even know if it causes any problems other than filling up the _internal log (There are noe issues with our environment relating to this warning as far as i know).

So i was wondering if anyone else have been experiencing these warnings, know what they are and know how to stop them? In peak search time there can be several million events per hour. 

One thing i have not yet tried, but will try as soon as possible, is to upgrade one of the standalone servers to Splunk 8.1.2 and see if that fixes things.

 

Labels (2)
0 Karma
1 Solution

mortf
Explorer

The server.conf i /etc/system/local is the same on all search heads and there are no specific site settings there, so all of them should follow the default site settings from the /etc/system/default server.conf

I asked this same question in slack and someone there told me that the issue i'm experiencing is a bug in the current major release version of Splunk. The issue will be fixed in the next major release version and so i'm choosing this as the answer to my question. There does not seem to be a viable workaround.

View solution in original post

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Check the site settings in server.conf on all search heads.

---
If this reply helps you, an upvote would be appreciated.
0 Karma

mortf
Explorer

The server.conf i /etc/system/local is the same on all search heads and there are no specific site settings there, so all of them should follow the default site settings from the /etc/system/default server.conf

I asked this same question in slack and someone there told me that the issue i'm experiencing is a bug in the current major release version of Splunk. The issue will be fixed in the next major release version and so i'm choosing this as the answer to my question. There does not seem to be a viable workaround.

View solution in original post

0 Karma
.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!