Splunk Search

Warning in internal log: unable to parse site_label

mortf
Explorer

I recently noticed a huge amount of warnings in the _internal logs for our search heads. events are all like this:

02-04-2021 12:22:08.485 +0300 WARN  SearchResultsFiles - Unable to parse site_label, label=invalid due to err="Invalid site id: invalid"

We are running a distributed environment with a search head cluster and all installations are Splunk 8.1.1. The warnings are logged only on the search heads.

When investigating i see this has occured for quite som time but i'm very qurious as to what this means.  There are no other indications in the _internal log that hints to why this warning keep appearing. I have however discovered that it seems to maybe be related to lookups and perhaps the kvstore. The reason i think so is that i can't force this warning when doing normal searches, but when i open dashbords that uses searches with macros and lookups they appear immediately. I've tried several different dashboards and searches and it seems consistent that anything with a lookup will produce this warning. 

I'm further thinking this might have happened when we upgraded to Splunk 8.1.1 recently. I've got two standalone servers for test purposes where one is running Splunk 8.1.1 and the other one is running Splunk 8.1.0.1
I have not been able to force this warning on the Splunk instance running 8.1.0.1 as of yet, but the one running 8.1.1 will have these warnings when i open dashbords and advanced searches.

I have not found anything in the Splunk "known issues" about this warning specifically. I don't even know if it causes any problems other than filling up the _internal log (There are noe issues with our environment relating to this warning as far as i know).

So i was wondering if anyone else have been experiencing these warnings, know what they are and know how to stop them? In peak search time there can be several million events per hour. 

One thing i have not yet tried, but will try as soon as possible, is to upgrade one of the standalone servers to Splunk 8.1.2 and see if that fixes things.

 

Labels (2)
0 Karma
1 Solution

mortf
Explorer

The server.conf i /etc/system/local is the same on all search heads and there are no specific site settings there, so all of them should follow the default site settings from the /etc/system/default server.conf

I asked this same question in slack and someone there told me that the issue i'm experiencing is a bug in the current major release version of Splunk. The issue will be fixed in the next major release version and so i'm choosing this as the answer to my question. There does not seem to be a viable workaround.

View solution in original post

0 Karma

dfronck
Communicator

V8.2.6 - Still not fixed.

https://docs.splunk.com/Documentation/Splunk/8.2.6/ReleaseNotes/Knownissues 

SPL-212495, SPL-196040, SPL-219811
Excessive logging 'WARN SearchResultsFiles Unable to parse site_label, label=invalid due to err="Invalid site id: invalid"' for SearchResultsFiles
Workaround: none
0 Karma

vgrote
Path Finder

Plus ca change ...

https://docs.splunk.com/Documentation/Splunk/9.0.0/ReleaseNotes/Knownissues

2021-09-22SPL-212495, SPL-196040, SPL-219811Excessive logging 'WARN SearchResultsFiles Unable to parse site_label, label=invalid due to err="Invalid site id: invalid"' for SearchResultsFiles

Workaround:
none
0 Karma

richgalloway
SplunkTrust
SplunkTrust

Check the site settings in server.conf on all search heads.

---
If this reply helps you, Karma would be appreciated.
0 Karma

mortf
Explorer

The server.conf i /etc/system/local is the same on all search heads and there are no specific site settings there, so all of them should follow the default site settings from the /etc/system/default server.conf

I asked this same question in slack and someone there told me that the issue i'm experiencing is a bug in the current major release version of Splunk. The issue will be fixed in the next major release version and so i'm choosing this as the answer to my question. There does not seem to be a viable workaround.

0 Karma

orion44
Communicator

@mortf I'm also experiencing this issue, the following error message is flooding my splunkd.log:

WARN SearchResultsFiles - Unable to parse site_label, label=invalid due to err="Invalid site id: invalid"

 

Can you confirm upgrading to a later version of Splunk resolved your issue? If so, what version did you upgrade to?

 

0 Karma

mortf
Explorer

Hi.

Upgrading to a new version of Splunk did not resolve this issue. The newest splunk version has this listet under known issues as well. Check out SPL-212495 and SPL-196040.

0 Karma
Get Updates on the Splunk Community!

How I Instrumented a Rust Application Without Knowing Rust

As a technical writer, I often have to edit or create code snippets for Splunk's distributions of ...

Splunk Community Platform Survey

Hey Splunk Community, Starting today, the community platform may prompt you to participate in a survey. The ...

Observability Highlights | November 2022 Newsletter

 November 2022Observability CloudEnd Of Support Extension for SignalFx Smart AgentSplunk is extending the End ...