Splunk Search

Warning in internal log: unable to parse site_label

mortf
Explorer

I recently noticed a huge amount of warnings in the _internal logs for our search heads. events are all like this:

02-04-2021 12:22:08.485 +0300 WARN  SearchResultsFiles - Unable to parse site_label, label=invalid due to err="Invalid site id: invalid"

We are running a distributed environment with a search head cluster and all installations are Splunk 8.1.1. The warnings are logged only on the search heads.

When investigating i see this has occured for quite som time but i'm very qurious as to what this means.  There are no other indications in the _internal log that hints to why this warning keep appearing. I have however discovered that it seems to maybe be related to lookups and perhaps the kvstore. The reason i think so is that i can't force this warning when doing normal searches, but when i open dashbords that uses searches with macros and lookups they appear immediately. I've tried several different dashboards and searches and it seems consistent that anything with a lookup will produce this warning. 

I'm further thinking this might have happened when we upgraded to Splunk 8.1.1 recently. I've got two standalone servers for test purposes where one is running Splunk 8.1.1 and the other one is running Splunk 8.1.0.1
I have not been able to force this warning on the Splunk instance running 8.1.0.1 as of yet, but the one running 8.1.1 will have these warnings when i open dashbords and advanced searches.

I have not found anything in the Splunk "known issues" about this warning specifically. I don't even know if it causes any problems other than filling up the _internal log (There are noe issues with our environment relating to this warning as far as i know).

So i was wondering if anyone else have been experiencing these warnings, know what they are and know how to stop them? In peak search time there can be several million events per hour. 

One thing i have not yet tried, but will try as soon as possible, is to upgrade one of the standalone servers to Splunk 8.1.2 and see if that fixes things.

 

Labels (2)
0 Karma
1 Solution

mortf
Explorer

The server.conf i /etc/system/local is the same on all search heads and there are no specific site settings there, so all of them should follow the default site settings from the /etc/system/default server.conf

I asked this same question in slack and someone there told me that the issue i'm experiencing is a bug in the current major release version of Splunk. The issue will be fixed in the next major release version and so i'm choosing this as the answer to my question. There does not seem to be a viable workaround.

View solution in original post

0 Karma

sansay1
Explorer

This appears to still not have been fixed, even though we are at version 8.2.7.
This is very disappointing from Splunk as it fills our logs with garbage and important logs get pushed out of the historical records. 
Splunk engineers: This needs to be fixed now.

0 Karma

dfronck
Communicator

V8.2.6 - Still not fixed.

https://docs.splunk.com/Documentation/Splunk/8.2.6/ReleaseNotes/Knownissues 

SPL-212495, SPL-196040, SPL-219811
Excessive logging 'WARN SearchResultsFiles Unable to parse site_label, label=invalid due to err="Invalid site id: invalid"' for SearchResultsFiles
Workaround: none
0 Karma

vgrote
Path Finder

Plus ca change ...

https://docs.splunk.com/Documentation/Splunk/9.0.0/ReleaseNotes/Knownissues

2021-09-22SPL-212495, SPL-196040, SPL-219811Excessive logging 'WARN SearchResultsFiles Unable to parse site_label, label=invalid due to err="Invalid site id: invalid"' for SearchResultsFiles

Workaround:
none
0 Karma

richgalloway
SplunkTrust
SplunkTrust

Check the site settings in server.conf on all search heads.

---
If this reply helps you, Karma would be appreciated.
0 Karma

mortf
Explorer

The server.conf i /etc/system/local is the same on all search heads and there are no specific site settings there, so all of them should follow the default site settings from the /etc/system/default server.conf

I asked this same question in slack and someone there told me that the issue i'm experiencing is a bug in the current major release version of Splunk. The issue will be fixed in the next major release version and so i'm choosing this as the answer to my question. There does not seem to be a viable workaround.

0 Karma

orion44
Communicator

@mortf I'm also experiencing this issue, the following error message is flooding my splunkd.log:

WARN SearchResultsFiles - Unable to parse site_label, label=invalid due to err="Invalid site id: invalid"

 

Can you confirm upgrading to a later version of Splunk resolved your issue? If so, what version did you upgrade to?

 

0 Karma

mortf
Explorer

Hi.

Upgrading to a new version of Splunk did not resolve this issue. The newest splunk version has this listet under known issues as well. Check out SPL-212495 and SPL-196040.

0 Karma
Get Updates on the Splunk Community!

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...

Splunkbase | Splunk Dashboard Examples App for SimpleXML End of Life

The Splunk Dashboard Examples App for SimpleXML will reach end of support on Dec 19, 2024, after which no new ...

Understanding Generative AI Techniques and Their Application in Cybersecurity

Watch On-Demand Artificial intelligence is the talk of the town nowadays, with industries of all kinds ...