Splunk Enterprise

Why is there splunk_instrumentation error after upgrade to Splunk Enterprise 9.0.2?

joshiro
Communicator

After upgrading Splunk Enterprise to 9.0.2 we are encountering the following error on every restart on CLI:

 

Checking conf files for problems...
                Invalid key in stanza [instrumentation.usage.tlsBestPractices] in /opt/splunk/etc/apps/splunk_instrumentation/default/savedsearches.conf, line 451: | append [| rest /services/configs/conf-pythonSslClientConfig | eval sslVerifyServerCert (value: if(isnull(sslVerifyServerCert),"unset",sslVerifyServerCert), splunk_server=sha256(splunk_server) | stats values(eai:acl.app) as python_configuredApp values(sslVerifyServerCert) as python_sslVerifyServerCert by splunk_server | eval python_configuredSystem=if(python_configuredApp="system","true","false") | fields python_sslVerifyServerCert, splunk_server, python_configuredSystem] 
| append [| rest /services/configs/conf-web/settings | eval mgmtHostPort=if(isnull(mgmtHostPort),"unset",mgmtHostPort), splunk_server=sha256(splunk_server) | stats values(eai:acl.app) as fwdrMgmtHostPort_configuredApp values(mgmtHostPort) as fwdr_mgmtHostPort by splunk_server | eval fwdrMgmtHostPort_configuredSystem=if(fwdrMgmtHostPort_configuredApp="system","true","false") | fields fwdrMgmtHostPort_sslVerifyServerCert, splunk_server, fwdrMgmtHostPort_configuredSystem] 
| append [| rest /services/configs/conf-server/sslConfig | eval cliVerifyServerName=if(isnull(cliVerifyServerName),"feature",cliVerifyServerName), splunk_server=sha256(splunk_server) | stats values(cliVerifyServerName) as servername_cliVerifyServerName values(eai:acl.app) as servername_configuredApp by splunk_server | eval cli_configuredSystem=if(cli_configuredApp="system","true","false") | fields cli_sslVerifyServerCert, splunk_server, cli_configuredSystem] 
| stats values(*) as * by splunk_server | eval date=now() | makejson output=data | eval _time=date, date=strftime(date,"%Y-%m-%d") | fields data date _time).
                Your indexes and inputs configurations are not internally consistent. For more information, run 'splunk btool check --debug'

 


This was not happening on 9.0.1 so we checked the 'savedsearches.conf' of the splunk_instrumentation app in the 9.0.1 tar and we found that the 9.0.2 'savedsearches.conf' is actually older and different than the 9.0.1 version.

 

~/Downloads$ diff savedsearches.conf.901 savedsearches.conf.902 | cat -A
447c447$
< | append [| rest /services/configs/conf-server/sslConfig | eval sslVerifyServerCert=if(isnull(sslVerifyServerCert),"unset",sslVerifyServerCert), splunk_server=sha256(splunk_server) | stats values(eai:acl.app) as global_configuredApp values(sslVerifyServerCert) as global_sslVerifyServerCert by splunk_server | eval global_configuredSystem=if(global_configuredApp="system","true","false") | fields global_sslVerifyServerCert, splunk_server, global_configuredSystem] \$
---$
> | append [| rest /services/configs/conf-server/sslConfig | eval sslVerifyServerCert=if(isnull(sslVerifyServerCert),"unset",sslVerifyServerCert), splunk_server=sha256(splunk_server) | stats values(eai:acl.app) as global_configuredApp values(sslVerifyServerCert) as global_sslVerifyServerCert by splunk_server | eval global_configuredSystem=if(global_configuredApp="system","true","false") | fields global_sslVerifyServerCert, splunk_server, global_configuredSystem] \ $

 


The difference lies in the scaped end of line character at the end.

We also tried to run this search from the GUI and it raises an error confirming that the search is indeed broken:

joshiro_0-1668095422685.png

We "solved" it by using the 9.0.1 version in the local folder of the app splunk_instrumentation.

Has anyone found out if this broken search is affecting Splunk Enterprise usage in anyway?

Labels (2)
0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

Since it's just instrumentation it should have no effect on your usage of Splunk.

You should file a support request with Splunk.

---
If this reply helps you, Karma would be appreciated.

View solution in original post

wongki
Explorer

For anyone else that found this topic, here's what Splunk Support said:

There is an unexpected white space after "\" on line 447 of $SPLUNK_HOME/etc/apps/splunk_instrumentation/default/savedsearches.conf.

joshiro
Communicator

Yes, that is exactly the problem.
We ended up copying a working version of that savedsearch to the local directory of the splunk_instrumentation app. This makes the report run normally, but still shows errors on CLI because the file in default directory is still broken.

What workaround did you apply?

0 Karma

jotne
Builder

There are no quick fix for this.  Only Splunk can fix this, so we just have to ignore this message until fixed by Splunk.  Strange thing is that I do see this on a clean installed Splunk server.  Just restart from CLI and message shows up.  How come that Splunk did not see this???

 

Tags (1)
0 Karma

joshiro
Communicator

It is not the first time that they release a package with some issues.
Recently, there was an issue with some federated.conf parameters not having the respective spec file definition, and it kept showing a warning on CLI on every restart.

They might be having some issues with QA. We are also trying to use SSG and Splunk Mobile and we are encountering several issues that didnt get caught by QA.

0 Karma

wongki
Explorer

I wanted to upgrade from 8.2.3.2 to 8.2.9 (before 9.0.2) and is also getting the same error.

Checking conf files for problems...
Invalid key in stanza [instrumentation.usage.tlsBestPractices] in /opt/splunk/etc/apps/splunk_instrumentation/default/savedsearches.conf, line 451: | append [| rest /services/configs/conf-pythonSslClientConfig | eval sslVerifyServerCert (value: if(isnull(sslVerifyServerCert),"unset",sslVerifyServerCert), splunk_server=sha256(splunk_server) | stats values(eai:acl.app) as python_configuredApp values(sslVerifyServerCert) as python_sslVerifyServerCert by splunk_server | eval python_configuredSystem=if(python_configuredApp="system","true","false") | fields python_sslVerifyServerCert, splunk_server, python_configuredSystem]
| append [| rest /services/configs/conf-web/settings | eval mgmtHostPort=if(isnull(mgmtHostPort),"unset",mgmtHostPort), splunk_server=sha256(splunk_server) | stats values(eai:acl.app) as fwdrMgmtHostPort_configuredApp values(mgmtHostPort) as fwdr_mgmtHostPort by splunk_server | eval fwdrMgmtHostPort_configuredSystem=if(fwdrMgmtHostPort_configuredApp="system","true","false") | fields fwdrMgmtHostPort_sslVerifyServerCert, splunk_server, fwdrMgmtHostPort_configuredSystem]
| append [| rest /services/configs/conf-server/sslConfig | eval cliVerifyServerName=if(isnull(cliVerifyServerName),"feature",cliVerifyServerName), splunk_server=sha256(splunk_server) | stats values(cliVerifyServerName) as servername_cliVerifyServerName values(eai:acl.app) as servername_configuredApp by splunk_server | eval cli_configuredSystem=if(cli_configuredApp="system","true","false") | fields cli_sslVerifyServerCert, splunk_server, cli_configuredSystem]
| stats values(*) as * by splunk_server | eval date=now() | makejson output=data | eval _time=date, date=strftime(date,"%Y-%m-%d") | fields data date _time).
Your indexes and inputs configurations are not internally consistent. For more information, run 'splunk btool check --debug'

 

0 Karma

joshiro
Communicator

Can you run a diff between the default savedsearches.conf of the splunk_instrumentation in both tar files?
It might return that the files are identical, and you ll need to pipe "cat -A" to see hidden special chars.

Just run:

diff <old-savedsearches> <new-savedsearches> | cat -A

 
A workaround in the 9.0.2 version was to copy the [instrumentation.usage.tlsBestPractices] stanza from the old working file and paste it in the local folder in the app. So splunk_instrumentation uses the old version of that particular stanza. This ll still show the error on CLI because the default file its still broken, but if you run the report it works just fine.

Hope this helps.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Since it's just instrumentation it should have no effect on your usage of Splunk.

You should file a support request with Splunk.

---
If this reply helps you, Karma would be appreciated.

joshiro
Communicator

Thanks for the reply.

We already opened a support case, they are aware of this issue and working on a workaround.

esalesap
Path Finder

The line in the search ending with:
"global_configuredSystem] \ "
has a space character after the backslash at the end of the line, remove the space and all is well

joshiro
Communicator

We "solve" it fixing the conf in local so it doesnt go against the manifest.
It still raises the warning but the actual search works.

0 Karma

jotne
Builder

Splunk is the only who can fix this and they have confirmed a fix for this in 9.0.4

0 Karma

jotne
Builder

Its fixed in 9.0.4.

But form the change log, it was fixed just days after 9.0.3 was released.

2022-12-02SPL-233492, SPL-229404Invalid key in stanza instrumentation.usage.tlsBestPractices
0 Karma

joshiro
Communicator

Good to know, any ETA for this new version?

Tags (1)
0 Karma

rabbidroid
Path Finder

Fixed in 9.0.4

Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...