Why do some local.meta files have \x00 errors in splunkd.log on my heavy forwarder?

[lycollicott]

I find these messages in splunkd.log:

02-15-2017 13:34:04.437 -0500 WARN  IniFile - C:\Splunk\var\run\searchpeers\my_dmc_server-1487183641\apps\fire_brigade\metadata\local.meta, line 4: Cannot parse into key-value pair: \x00

\x00 is a HEX 0, so why is it in my search bundle's local.meta?

NOTE: There is not \x00 in the app's original local.meta file on the DMC.

[timpacl]

I have this issue across thousands of local.meta files in v8.0.3. Was the issue reintroduced.

[jcrabb_splunk]

Just a quick follow up. We suspect that it is caused by the tar format change introduced in a previous version. We are not aware of any problems this is causing outside of the WARN messages. I'll follow up again once more information has been provided.

[jcrabb_splunk · Accepted solution]

I am not sure what the cause is. I had seen this on a few cases recently where we saw those types of errors and had speculated on what may be happening. As it wasn't the reason for the case, we did not deep dive and chalked it up to a fluke. When I saw your answers post it made me consider that it may be a larger problem. Going through the current cases, I am seeing a few with the same local.meta errors. As I have data in hand, I am going to file a bug with our dev team to see if they understand what is happening. As soon as I get more information, I will share that with you.

## EDIT ##

This is fixed in:

• 6.6.11 (SPL-136970)
• 7.0.6 (SPL-158503)
• 7.1.3 (SPL-158504)

[goelli]

Is there any news on this?
I have the same errors since updating from 6.4.4 to 6.5.3

[moesaidi]

Similar issue in 7.0.0:
08-16-2018 10:35:01.688 -0300 WARN IniFile - C:\PATH\searchpeers\NAME-ID\system\metadata\local.meta, line 22: Cannot parse into key-value pair:

(it shows "NULL")
Is this the same bug? similar bug? new bug?

[jcrabb_splunk]

Appears to be the same. For 7.0.x it is fixed in 7.0.6 (SPL-158503).

[splunkitsicherh]

Don´t know exactly how it was fixed but we run SPLUNK 7.1.1 and there´s the Key-value ERROR in a lot of local.meta files. splunk enterprise runs on linux system(s)

OK I find the SPL-136970 and SPL-158504 in latest-version (7.1.3) on Website. so perhaps with this version the bug is fixed.

Is there a way to only get this bugfix?

[jcrabb_splunk]

We do not offer single issue patches. You will want to upgrade when its convenient.

[jcrabb_splunk]

Good news, this finally been tracked down and is going through Q/A. We expect the bug to be addressed in 6.6.10 and I would assume it will be ported to the other, newer versions.

[jralston]

I am also seeing similar errors. I assume it is also causing of my bundle replication issues. Hope for an update on this.

[jcrabb_splunk]

It does not cause any problems other than this WARN message.

[jcrabb_splunk]

No new info just yet. There is a bug and it's being investigated. If you want to suppress the message, edit $SPLUNK_HOME/etc/log-cmdline.cfg, locate the following line:

category.IniFile=INFO

and change it to ERROR:

category.IniFile=ERROR

[martinho]

Why have you recommended using log-cmdline.cfg instead of log-local.cfg (persists over Splunk upgrade) or log.cfg? I'm just curious - I have used log-local.cfg and it seems to work.

[goelli]

Thanks. Can you keep us informed about the bug status here or should I open a case?

You workaround is a good idea, but it's not working for us. If I switch log level to ERROR, I will not receive correct warnings anymore.
We need the warnings to see, if someone forgets the line-breaking marker "\" if a search or any other value is using multiple lines. In worst case we have a saved search which is missing some lines, because of missing "\", which might result in a security event not recognized...

[ChrisBell04]

I went ahead and created a case (#507968 local.meta - Cannot parse into key-value pair) and was told engineering already aware.

SPL-136970 - default and local meta files getting corrupt or being altered in such a way as to cause warnings.
for those who want to track it.

[jcrabb_splunk]

Yes, I will keep you updated.