Splunk Enterprise

Error in savedsearches.conf- Invalid key in stanza - splunk_instrumentation - savedseaches.conf v8.2.9

rvany
Communicator

As my original subject led to some weird error message about message flooding - here it is again:

Subject: Invalid key in stanza - splunk_instrumentation - savedseaches.conf v8.2.9

Version 8.2.9 (Linux, tgz-version) brings the "Invalid key in stanza" error in line 451 of `/opt/splunk/etc/apps/splunk_instrumentation/default/savedsearches.conf` - file. This wasn't the case in v.8.27.

It turns out that the named file differs in one character between the two versions:

A space added after the "\" (for line continuation) in v8.2.9. After removing that single space the `splunk restart` command run through without errors.

Labels (1)
Tags (1)

manuelostertag
Path Finder

@rvanythanks for the community entry, this has taken away the doubts about ourselves 🙂

And first thought something went wrong during installation on our system, but we found the "problem" on all of our server.

With a patch (bash script) we fixed the "typo" (we don't know if it has a negative impact on Splunk's behaviour or it is just an typo)  and change the sha256sum of the file in the manifest file so that no error message comes up when Splunk starts.

This should have been noticed by Splunk when testing the software.

0 Karma

manuelostertag
Path Finder

Luckily, the error is solved in 8.2.10 😅

0 Karma

dfgrtKJH
Path Finder

Same Problem in 9.0.2:

/opt/splunk/bin/splunk btool check --debug

Checking: /opt/splunk/etc/apps/splunk_instrumentation/default/savedsearches.conf
		Invalid key in stanza [instrumentation.usage.tlsBestPractices] in /opt/splunk/etc/apps/splunk_instrumentation/default/savedsearches.conf, line 451: | append [| rest /services/configs/conf-pythonSslClientConfig | eval sslVerifyServerCert (value: if(isnull(sslVerifyServerCert),"unset",sslVerifyServerCert), splunk_server=sha256(splunk_server) | stats values(eai:acl.app) as python_configuredApp values(sslVerifyServerCert) as python_sslVerifyServerCert by splunk_server | eval python_configuredSystem=if(python_configuredApp="system","true","false") | fields python_sslVerifyServerCert, splunk_server, python_configuredSystem] 
| append [| rest /services/configs/conf-web/settings | eval mgmtHostPort=if(isnull(mgmtHostPort),"unset",mgmtHostPort), splunk_server=sha256(splunk_server) | stats values(eai:acl.app) as fwdrMgmtHostPort_configuredApp values(mgmtHostPort) as fwdr_mgmtHostPort by splunk_server | eval fwdrMgmtHostPort_configuredSystem=if(fwdrMgmtHostPort_configuredApp="system","true","false") | fields fwdrMgmtHostPort_sslVerifyServerCert, splunk_server, fwdrMgmtHostPort_configuredSystem] 
| append [| rest /services/configs/conf-server/sslConfig | eval cliVerifyServerName=if(isnull(cliVerifyServerName),"feature",cliVerifyServerName), splunk_server=sha256(splunk_server) | stats values(cliVerifyServerName) as servername_cliVerifyServerName values(eai:acl.app) as servername_configuredApp by splunk_server | eval cli_configuredSystem=if(cli_configuredApp="system","true","false") | fields cli_sslVerifyServerCert, splunk_server, cli_configuredSystem] 
| stats values(*) as * by splunk_server | eval date=now() | makejson output=data | eval _time=date, date=strftime(date,"%Y-%m-%d") | fields data date _time).
0 Karma

dfgrtKJH
Path Finder
  • In 9.0.2, I changed /opt/splunk/etc/apps/splunk_instrumentation/default/savedsearches.conf line 447 and removed the space at the end of the line after the "\" character.
  • sha256sum /opt/splunk/etc/apps/splunk_instrumentation/default/savedsearches.conf
    e00229cf2b4fee8ecf2232d98358d1a32563bb7edf6a60ec2274e765fb51e22d
  • edit /opt/splunk/splunk-9.0.2-17e00c557dc1-linux-2.6-x86_64-manifest
  • restart splunk

Problem solved. The changed file is now identical to the same file in version 9.0.1.

@splunkPlease fix this typo in the file

jerryz_splunk
Splunk Employee
Splunk Employee

Thank you for the feedback. We have an internal bug number to address and fix it now.

Tags (1)
0 Karma

dfgrtKJH
Path Finder

This is not fixed in the new version 9.0.3.

Very sad...

0 Karma

jerryz_splunk
Splunk Employee
Splunk Employee

The fix of this is scheduled in 9.0.4. Thanks for yor patience. 🙂

0 Karma

sistemistiposta
Path Finder

Yeah it works!

Not good that Splunk distributes bugged packages.

Thank you @dfgrtKJH !

 

   Marco

0 Karma

rvany
Communicator

Yes, that's exactly the line for th 8.2.9 version although the sha256sum is different there.

0 Karma

jeffh-cf
Engager

I didn't see mention of the issue in 9.0.3 but I can confirm that the issue is also in 9.0.3. 

I implemented the recommended change and it resolved the error.

sha256sum shows the following after making the change:

e00229cf2b4fee8ecf2232d98358d1a32563bb7edf6a60ec2274e765fb51e22d  savedsearches.conf

Thanks all! I'm glad I wasn't the only one running into this. 🙂

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...