Why is there splunk_instrumentation error after upgrade to Splunk Enterprise 9.0.2?

[joshiro]

After upgrading Splunk Enterprise to 9.0.2 we are encountering the following error on every restart on CLI:

Checking conf files for problems...
                Invalid key in stanza [instrumentation.usage.tlsBestPractices] in /opt/splunk/etc/apps/splunk_instrumentation/default/savedsearches.conf, line 451: | append [| rest /services/configs/conf-pythonSslClientConfig | eval sslVerifyServerCert (value: if(isnull(sslVerifyServerCert),"unset",sslVerifyServerCert), splunk_server=sha256(splunk_server) | stats values(eai:acl.app) as python_configuredApp values(sslVerifyServerCert) as python_sslVerifyServerCert by splunk_server | eval python_configuredSystem=if(python_configuredApp="system","true","false") | fields python_sslVerifyServerCert, splunk_server, python_configuredSystem] 
| append [| rest /services/configs/conf-web/settings | eval mgmtHostPort=if(isnull(mgmtHostPort),"unset",mgmtHostPort), splunk_server=sha256(splunk_server) | stats values(eai:acl.app) as fwdrMgmtHostPort_configuredApp values(mgmtHostPort) as fwdr_mgmtHostPort by splunk_server | eval fwdrMgmtHostPort_configuredSystem=if(fwdrMgmtHostPort_configuredApp="system","true","false") | fields fwdrMgmtHostPort_sslVerifyServerCert, splunk_server, fwdrMgmtHostPort_configuredSystem] 
| append [| rest /services/configs/conf-server/sslConfig | eval cliVerifyServerName=if(isnull(cliVerifyServerName),"feature",cliVerifyServerName), splunk_server=sha256(splunk_server) | stats values(cliVerifyServerName) as servername_cliVerifyServerName values(eai:acl.app) as servername_configuredApp by splunk_server | eval cli_configuredSystem=if(cli_configuredApp="system","true","false") | fields cli_sslVerifyServerCert, splunk_server, cli_configuredSystem] 
| stats values(*) as * by splunk_server | eval date=now() | makejson output=data | eval _time=date, date=strftime(date,"%Y-%m-%d") | fields data date _time).
                Your indexes and inputs configurations are not internally consistent. For more information, run 'splunk btool check --debug'

This was not happening on 9.0.1 so we checked the 'savedsearches.conf' of the splunk_instrumentation app in the 9.0.1 tar and we found that the 9.0.2 'savedsearches.conf' is actually older and different than the 9.0.1 version.

~/Downloads$ diff savedsearches.conf.901 savedsearches.conf.902 | cat -A
447c447$
< | append [| rest /services/configs/conf-server/sslConfig | eval sslVerifyServerCert=if(isnull(sslVerifyServerCert),"unset",sslVerifyServerCert), splunk_server=sha256(splunk_server) | stats values(eai:acl.app) as global_configuredApp values(sslVerifyServerCert) as global_sslVerifyServerCert by splunk_server | eval global_configuredSystem=if(global_configuredApp="system","true","false") | fields global_sslVerifyServerCert, splunk_server, global_configuredSystem] \$
---$
> | append [| rest /services/configs/conf-server/sslConfig | eval sslVerifyServerCert=if(isnull(sslVerifyServerCert),"unset",sslVerifyServerCert), splunk_server=sha256(splunk_server) | stats values(eai:acl.app) as global_configuredApp values(sslVerifyServerCert) as global_sslVerifyServerCert by splunk_server | eval global_configuredSystem=if(global_configuredApp="system","true","false") | fields global_sslVerifyServerCert, splunk_server, global_configuredSystem] \ $

The difference lies in the scaped end of line character at the end.

We also tried to run this search from the GUI and it raises an error confirming that the search is indeed broken:

We "solved" it by using the 9.0.1 version in the local folder of the app splunk_instrumentation.

Has anyone found out if this broken search is affecting Splunk Enterprise usage in anyway?

[wongki]

For anyone else that found this topic, here's what Splunk Support said:

There is an unexpected white space after "\" on line 447 of $SPLUNK_HOME/etc/apps/splunk_instrumentation/default/savedsearches.conf.

[joshiro]

Yes, that is exactly the problem.
We ended up copying a working version of that savedsearch to the local directory of the splunk_instrumentation app. This makes the report run normally, but still shows errors on CLI because the file in default directory is still broken.

What workaround did you apply?

[jotne]

There are no quick fix for this.  Only Splunk can fix this, so we just have to ignore this message until fixed by Splunk.  Strange thing is that I do see this on a clean installed Splunk server.  Just restart from CLI and message shows up.  How come that Splunk did not see this???

[joshiro]

It is not the first time that they release a package with some issues.
Recently, there was an issue with some federated.conf parameters not having the respective spec file definition, and it kept showing a warning on CLI on every restart.

They might be having some issues with QA. We are also trying to use SSG and Splunk Mobile and we are encountering several issues that didnt get caught by QA.

[wongki]

I wanted to upgrade from 8.2.3.2 to 8.2.9 (before 9.0.2) and is also getting the same error.

Checking conf files for problems...
Invalid key in stanza [instrumentation.usage.tlsBestPractices] in /opt/splunk/etc/apps/splunk_instrumentation/default/savedsearches.conf, line 451: | append [| rest /services/configs/conf-pythonSslClientConfig | eval sslVerifyServerCert (value: if(isnull(sslVerifyServerCert),"unset",sslVerifyServerCert), splunk_server=sha256(splunk_server) | stats values(eai:acl.app) as python_configuredApp values(sslVerifyServerCert) as python_sslVerifyServerCert by splunk_server | eval python_configuredSystem=if(python_configuredApp="system","true","false") | fields python_sslVerifyServerCert, splunk_server, python_configuredSystem]
| append [| rest /services/configs/conf-web/settings | eval mgmtHostPort=if(isnull(mgmtHostPort),"unset",mgmtHostPort), splunk_server=sha256(splunk_server) | stats values(eai:acl.app) as fwdrMgmtHostPort_configuredApp values(mgmtHostPort) as fwdr_mgmtHostPort by splunk_server | eval fwdrMgmtHostPort_configuredSystem=if(fwdrMgmtHostPort_configuredApp="system","true","false") | fields fwdrMgmtHostPort_sslVerifyServerCert, splunk_server, fwdrMgmtHostPort_configuredSystem]
| append [| rest /services/configs/conf-server/sslConfig | eval cliVerifyServerName=if(isnull(cliVerifyServerName),"feature",cliVerifyServerName), splunk_server=sha256(splunk_server) | stats values(cliVerifyServerName) as servername_cliVerifyServerName values(eai:acl.app) as servername_configuredApp by splunk_server | eval cli_configuredSystem=if(cli_configuredApp="system","true","false") | fields cli_sslVerifyServerCert, splunk_server, cli_configuredSystem]
| stats values(*) as * by splunk_server | eval date=now() | makejson output=data | eval _time=date, date=strftime(date,"%Y-%m-%d") | fields data date _time).
Your indexes and inputs configurations are not internally consistent. For more information, run 'splunk btool check --debug'

[joshiro]

Can you run a diff between the default savedsearches.conf of the splunk_instrumentation in both tar files?
It might return that the files are identical, and you ll need to pipe "cat -A" to see hidden special chars.

Just run:

diff <old-savedsearches> <new-savedsearches> | cat -A

 
A workaround in the 9.0.2 version was to copy the [instrumentation.usage.tlsBestPractices] stanza from the old working file and paste it in the local folder in the app. So splunk_instrumentation uses the old version of that particular stanza. This ll still show the error on CLI because the default file its still broken, but if you run the report it works just fine.

Hope this helps.

[richgalloway · Accepted solution]

Since it's just instrumentation it should have no effect on your usage of Splunk.

You should file a support request with Splunk.

[joshiro]

Thanks for the reply.

We already opened a support case, they are aware of this issue and working on a workaround.

[esalesap]

The line in the search ending with:
"global_configuredSystem] \ "
has a space character after the backslash at the end of the line, remove the space and all is well

[joshiro]

We "solve" it fixing the conf in local so it doesnt go against the manifest.
It still raises the warning but the actual search works.

[jotne]

Splunk is the only who can fix this and they have confirmed a fix for this in 9.0.4

[jotne]

Its fixed in 9.0.4.

But form the change log, it was fixed just days after 9.0.3 was released.

2022-12-02

SPL-233492, SPL-229404

Invalid key in stanza instrumentation.usage.tlsBestPractices

[joshiro]

Good to know, any ETA for this new version?

[rabbidroid]

Fixed in 9.0.4