Error in savedsearches.conf- Invalid key in stanza - splunk_instrumentation - savedseaches.conf v8.2.9

[rvany]

As my original subject led to some weird error message about message flooding - here it is again:

Subject: Invalid key in stanza - splunk_instrumentation - savedseaches.conf v8.2.9

Version 8.2.9 (Linux, tgz-version) brings the "Invalid key in stanza" error in line 451 of `/opt/splunk/etc/apps/splunk_instrumentation/default/savedsearches.conf` - file. This wasn't the case in v.8.27.

It turns out that the named file differs in one character between the two versions:

A space added after the "\" (for line continuation) in v8.2.9. After removing that single space the `splunk restart` command run through without errors.

[manuelostertag]

@rvanythanks for the community entry, this has taken away the doubts about ourselves 🙂

And first thought something went wrong during installation on our system, but we found the "problem" on all of our server.

With a patch (bash script) we fixed the "typo" (we don't know if it has a negative impact on Splunk's behaviour or it is just an typo)  and change the sha256sum of the file in the manifest file so that no error message comes up when Splunk starts.

This should have been noticed by Splunk when testing the software.

[manuelostertag]

Luckily, the error is solved in 8.2.10 😅

[dfgrtKJH]

Same Problem in 9.0.2:

/opt/splunk/bin/splunk btool check --debug

Checking: /opt/splunk/etc/apps/splunk_instrumentation/default/savedsearches.conf
		Invalid key in stanza [instrumentation.usage.tlsBestPractices] in /opt/splunk/etc/apps/splunk_instrumentation/default/savedsearches.conf, line 451: | append [| rest /services/configs/conf-pythonSslClientConfig | eval sslVerifyServerCert (value: if(isnull(sslVerifyServerCert),"unset",sslVerifyServerCert), splunk_server=sha256(splunk_server) | stats values(eai:acl.app) as python_configuredApp values(sslVerifyServerCert) as python_sslVerifyServerCert by splunk_server | eval python_configuredSystem=if(python_configuredApp="system","true","false") | fields python_sslVerifyServerCert, splunk_server, python_configuredSystem] 
| append [| rest /services/configs/conf-web/settings | eval mgmtHostPort=if(isnull(mgmtHostPort),"unset",mgmtHostPort), splunk_server=sha256(splunk_server) | stats values(eai:acl.app) as fwdrMgmtHostPort_configuredApp values(mgmtHostPort) as fwdr_mgmtHostPort by splunk_server | eval fwdrMgmtHostPort_configuredSystem=if(fwdrMgmtHostPort_configuredApp="system","true","false") | fields fwdrMgmtHostPort_sslVerifyServerCert, splunk_server, fwdrMgmtHostPort_configuredSystem] 
| append [| rest /services/configs/conf-server/sslConfig | eval cliVerifyServerName=if(isnull(cliVerifyServerName),"feature",cliVerifyServerName), splunk_server=sha256(splunk_server) | stats values(cliVerifyServerName) as servername_cliVerifyServerName values(eai:acl.app) as servername_configuredApp by splunk_server | eval cli_configuredSystem=if(cli_configuredApp="system","true","false") | fields cli_sslVerifyServerCert, splunk_server, cli_configuredSystem] 
| stats values(*) as * by splunk_server | eval date=now() | makejson output=data | eval _time=date, date=strftime(date,"%Y-%m-%d") | fields data date _time).

[dfgrtKJH]

• In 9.0.2, I changed /opt/splunk/etc/apps/splunk_instrumentation/default/savedsearches.conf line 447 and removed the space at the end of the line after the "\" character.
• sha256sum /opt/splunk/etc/apps/splunk_instrumentation/default/savedsearches.conf
  e00229cf2b4fee8ecf2232d98358d1a32563bb7edf6a60ec2274e765fb51e22d
• edit /opt/splunk/splunk-9.0.2-17e00c557dc1-linux-2.6-x86_64-manifest
• restart splunk

Problem solved. The changed file is now identical to the same file in version 9.0.1.

@splunkPlease fix this typo in the file

[jerryz_splunk]

Thank you for the feedback. We have an internal bug number to address and fix it now.

[dfgrtKJH]

This is not fixed in the new version 9.0.3.

Very sad...

[jerryz_splunk]

The fix of this is scheduled in 9.0.4. Thanks for yor patience. 🙂

[sistemistiposta]

Yeah it works!

Not good that Splunk distributes bugged packages.

Thank you @dfgrtKJH !

   Marco

[rvany]

Yes, that's exactly the line for th 8.2.9 version although the sha256sum is different there.

[jeffh-cf]

I didn't see mention of the issue in 9.0.3 but I can confirm that the issue is also in 9.0.3. 

I implemented the recommended change and it resolved the error.

sha256sum shows the following after making the change:

e00229cf2b4fee8ecf2232d98358d1a32563bb7edf6a60ec2274e765fb51e22d  savedsearches.conf

Thanks all! I'm glad I wasn't the only one running into this. 🙂