- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Error in savedsearches.conf- Invalid key in stanza - splunk_instrumentation - savedseaches.conf v8.2.9

As my original subject led to some weird error message about message flooding - here it is again:
Subject: Invalid key in stanza - splunk_instrumentation - savedseaches.conf v8.2.9
Version 8.2.9 (Linux, tgz-version) brings the "Invalid key in stanza" error in line 451 of `/opt/splunk/etc/apps/splunk_instrumentation/default/savedsearches.conf` - file. This wasn't the case in v.8.27.
It turns out that the named file differs in one character between the two versions:
A space added after the "\" (for line continuation) in v8.2.9. After removing that single space the `splunk restart` command run through without errors.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@rvanythanks for the community entry, this has taken away the doubts about ourselves 🙂
And first thought something went wrong during installation on our system, but we found the "problem" on all of our server.
With a patch (bash script) we fixed the "typo" (we don't know if it has a negative impact on Splunk's behaviour or it is just an typo) and change the sha256sum of the file in the manifest file so that no error message comes up when Splunk starts.
This should have been noticed by Splunk when testing the software.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Luckily, the error is solved in 8.2.10 😅
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Same Problem in 9.0.2:
/opt/splunk/bin/splunk btool check --debug
Checking: /opt/splunk/etc/apps/splunk_instrumentation/default/savedsearches.conf
Invalid key in stanza [instrumentation.usage.tlsBestPractices] in /opt/splunk/etc/apps/splunk_instrumentation/default/savedsearches.conf, line 451: | append [| rest /services/configs/conf-pythonSslClientConfig | eval sslVerifyServerCert (value: if(isnull(sslVerifyServerCert),"unset",sslVerifyServerCert), splunk_server=sha256(splunk_server) | stats values(eai:acl.app) as python_configuredApp values(sslVerifyServerCert) as python_sslVerifyServerCert by splunk_server | eval python_configuredSystem=if(python_configuredApp="system","true","false") | fields python_sslVerifyServerCert, splunk_server, python_configuredSystem]
| append [| rest /services/configs/conf-web/settings | eval mgmtHostPort=if(isnull(mgmtHostPort),"unset",mgmtHostPort), splunk_server=sha256(splunk_server) | stats values(eai:acl.app) as fwdrMgmtHostPort_configuredApp values(mgmtHostPort) as fwdr_mgmtHostPort by splunk_server | eval fwdrMgmtHostPort_configuredSystem=if(fwdrMgmtHostPort_configuredApp="system","true","false") | fields fwdrMgmtHostPort_sslVerifyServerCert, splunk_server, fwdrMgmtHostPort_configuredSystem]
| append [| rest /services/configs/conf-server/sslConfig | eval cliVerifyServerName=if(isnull(cliVerifyServerName),"feature",cliVerifyServerName), splunk_server=sha256(splunk_server) | stats values(cliVerifyServerName) as servername_cliVerifyServerName values(eai:acl.app) as servername_configuredApp by splunk_server | eval cli_configuredSystem=if(cli_configuredApp="system","true","false") | fields cli_sslVerifyServerCert, splunk_server, cli_configuredSystem]
| stats values(*) as * by splunk_server | eval date=now() | makejson output=data | eval _time=date, date=strftime(date,"%Y-%m-%d") | fields data date _time).
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
- In 9.0.2, I changed /opt/splunk/etc/apps/splunk_instrumentation/default/savedsearches.conf line 447 and removed the space at the end of the line after the "\" character.
- sha256sum /opt/splunk/etc/apps/splunk_instrumentation/default/savedsearches.conf
e00229cf2b4fee8ecf2232d98358d1a32563bb7edf6a60ec2274e765fb51e22d - edit /opt/splunk/splunk-9.0.2-17e00c557dc1-linux-2.6-x86_64-manifest
- restart splunk
Problem solved. The changed file is now identical to the same file in version 9.0.1.
@splunkPlease fix this typo in the file
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content

Thank you for the feedback. We have an internal bug number to address and fix it now.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
This is not fixed in the new version 9.0.3.
Very sad...
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content

The fix of this is scheduled in 9.0.4. Thanks for yor patience. 🙂
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content

- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content

Yes, that's exactly the line for th 8.2.9 version although the sha256sum is different there.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I didn't see mention of the issue in 9.0.3 but I can confirm that the issue is also in 9.0.3.
I implemented the recommended change and it resolved the error.
sha256sum shows the following after making the change:
e00229cf2b4fee8ecf2232d98358d1a32563bb7edf6a60ec2274e765fb51e22d savedsearches.conf
Thanks all! I'm glad I wasn't the only one running into this. 🙂
