topic Re: Linear memory growth with Splunk 9.4.0 and above in Splunk Enterprise

Linear memory growth with Splunk 9.4.0 and above

hrawat — Wed, 26 Feb 2025 01:51:58 GMT

Linear memory growth on any splunk instance configured to receive data on splunktcpin, tcpin and udpin ports.

Following config in server.conf will fix the memory growth.

[prometheus]

disabled = true

Re: Linear memory growth with Splunk 9.4.0 and above

gjanders — Wed, 26 Feb 2025 02:20:11 GMT

I noticed that https://docs.splunk.com/Documentation/Splunk/9.4.0/Admin/Serverconf does not mention prometheus.

Is this an undocumented feature that is getting disabled to prevent a memory leak issue?

Re: Linear memory growth with Splunk 9.4.0 and above

hrawat — Wed, 26 Feb 2025 02:38:08 GMT

Although it was not documented but 9.3.x/9.2.x/9.1.x etc/system/default/server.conf you will find

[prometheus]
disabled = true

It was added in server.conf to prevent unwanted memory growth caused by prometheus. Unintentionally stanza got removed from 9.4.0. So you restore it back.

Re: Linear memory growth with Splunk 9.4.0 and above

jotne — Fri, 28 Feb 2025 10:20:23 GMT

Can confirm that it fixed memory leak that we see on our upgraded HF server.

This is not fixed in the newly released 9.4.1.

But what does prometheus do in splunk? Is it some new function that was added to the 9.x server and was set to disabled? Do not find any info in server.conf docs.

Re: Linear memory growth with Splunk 9.4.0 and above

hrawat — Fri, 28 Feb 2025 12:55:09 GMT

It's https://prometheus.io/ support that was added almost 4 years ago(8.2.0). But it was disabled due to memory explosion since 8.2.1

Re: Linear memory growth with Splunk 9.4.0 and above

jotne — Fri, 28 Feb 2025 11:27:35 GMT

4 years?????

And nothing has been done to fix it. This part should then be removed from the code then.

Here you see memory one of our HF whas upgraded from 9.3.2 to 9.4. When all memory are used up, it runs for some hour more and then dies. We reported this issue just some days after 9.4.0 was released, and did get the fix just now.

Re: Linear memory growth with Splunk 9.4.0 and above

AndyM — Wed, 07 May 2025 15:31:50 GMT

Is this issue likely to be fixed in an upcoming version release?

Re: Linear memory growth with Splunk 9.4.0 and above

jotne — Wed, 07 May 2025 15:58:14 GMT

I do not see any fix for this in the just released 9.4.2 that was released month after it was discovered in 9.4.0

There is a setting in the next beta for Splunk so maybe it will come in 9.4.3

Also strange that this setting is not mention in the latest documentation:
https://docs.splunk.com/Documentation/Splunk/latest/Admin/Serverconf

[prometheus]
disabled = true

Re: Linear memory growth with Splunk 9.4.0 and above

hrawat — Wed, 07 May 2025 16:38:35 GMT

It's not fixed in upcoming releases.
However the fix (whenever part of a release) will also be same as the workaround.

[prometheus]
disabled = true