Considerations regarding system-wide resource limits on *nix systems and data segment size (ulimit -d)

[yeahnah]

Hi Splunk Admins

Just looking for some advice around setting the data segment size (ulimit -d) in Splunk, on a Linux  server (RHEL). 

Older documentation (v7.3) recommended setting this value to basically be an unlimited size, with a Kibibyte value of 1073741824 or ~1TB. 
https://docs.splunk.com/Documentation/Splunk/7.3.8/Installation/Systemrequirements#Considerations_re...

Data segment size

ulimit -d

1073741824

I see the v8.x documentation has now changed the data segment size recommendation to be more a general guideline, with an 8GB example.

https://docs.splunk.com/Documentation/Splunk/latest/Installation/Systemrequirements#Considerations_r...

Data segment size

ulimit -d

The maximum RAM you want Splunk Enterprise to allocate in kilobytes. For example, 8GB is 8000000.

It appears Splunk do not really have a strong opinion on a minimum size now either.  I think on RHEV Linux, the data segment size just defaults to unlimited anyway, or at least on our VM servers it does.

I don't believe setting this value alone helps protect Splunk from excessive memory use either.  From what I can tell with googling about data segments, if it was indeed set to a value, then it does not even need to be set to an excessively large value.  Happy to admit I'm no expert though.

Anyway, just wondering if anyone has some experience with setting this value in their environments, or even a view if this data segment size limit even really needs to be set at all - on Linux at least.

[yeahnah · Accepted solution]

Will answer my own question.

I raised a case with Splunk and they basically came back and said they set their Linux servers to be unlimited for the data segment size value (ulimit -d unlimited).

# vi /etc/security/limits.conf
...<snip>...

# Data segment size: ulimit -d
splunk soft data unlimited
splunk hard data unlimited

Note, if using systemd then will be set under  /etc/systemd/system/Splunkd.service - refer to doc links in question section.

Not sure why Splunk docs don't just specify the same unlimited value, as the current recommendation is vague and more confusing than useful.