topic Re: limit the data size in Docker with maxTotalDataSizeMB does not work in Getting Data In

limit the data size in Docker with maxTotalDataSizeMB does not work

somoarn — Tue, 31 Aug 2021 14:20:30 GMT

I am learning Splunk and I have built the following test environment in Docker:

Splunk server running in a container, using the official docker image: image: splunk/splunk:8.2
I have another docker container, call it client where I installed the forwarder and then I added a file to monitor with the $SPLUNK_HOME/bin/splunk add monitor $MY_LOGFILE -index main -sourcetype mylog command.

Everything works fine. If I append $MY_LOGFILE in the client docker container with

echo "hello" >> $MY_LOGFILE

command then I can see the new line in the Splunk web console.

Now I am appending/feeding my log file with an endless bash counter-up loop and I can see everything in the Splunk web console. Great.

My question:

I would like to delete old records from Splunk to save disk space, so I followed the documentation and I did this:

sudo vi /opt/splunk/etc/system/local/indexes.conf

with this content

[main] maxTotalDataSizeMB=1 rozenTimePeriodInSecs=300 disabled=false

As I know this allows Splunk to automatically delete old data when my index hits the 1MB size. After I have created this new config file, I restarted the Splunk Docker container (and Splunk as well manually).

But actually, nothing happens. It seems that this setting is not considered, and I see the increasing number of records in the index and index size is also increasing without limitation in Splunk.

I use the following commands to check index size:

sourcetype=mylog | stats count as Records
index=_internal source=* type=Usage idx=* | eval SIZE=b/1024 | stats sum(SIZE) by st, result: 30756.775390625

But when I stop Splunk then I am able to clean up the index with this command:

splunk stop splunk clean eventdata splunk start

But I have a scenario where I need to limit the size of the index and the disk usage that is used by Splunk index "realtime", without stop and start.

What I am missing here?

Thx

Re: limit the data size in Docker with maxTotalDataSizeMB does not work

codebuilder — Tue, 31 Aug 2021 14:19:45 GMT

This could be caused by several issues.
The first to check is the ownership of your indexes.conf. I noticed you used sudo to edit it, and therefore it will be owned by root. Splunk runs as the "splunk" user by default. Because of that it will ignore the file you put into local because it doesn't have permission to read it. Unless you are explicitly running Splunk as root. Change the ownership of the file and all directories above.

Next would be restarting the container itself. Unless you have persistent storage configured, the container will pick a new directory for storage each time it is restarted.

There are other causes, but those are where I would start, and lean towards the ownership of indexes.conf.

Re: limit the data size in Docker with maxTotalDataSizeMB does not work

somoarn — Tue, 31 Aug 2021 15:12:50 GMT

Thx for the response. This is a good idea.

I have dropped my containers and started again from the scratch.

$ docker exec -u splunk -it splunk /bin/bash vi /opt/splunk/etc/system/local/indexes.conf [main] maxTotalDataSizeMB=1 frozenTimePeriodInSecs=300 disabled=false ls -all /opt/splunk/etc/system/local/indexes.conf -rw-rw-r-- 1 splunk splunk 71 Aug 31 14:58 /opt/splunk/etc/system/local/indexes.conf

Then I restarted the Splunk this way:

$ cd /opt/splunk/bin/ $ ./splunk restart Stopping splunkd... Shutting down. Please wait, as this may take a few minutes. ..................................... Stopping splunk helpers... Done. Splunk> Needle. Haystack. Found. Checking prerequisites... Checking http port [8000]: open Checking mgmt port [8089]: open Checking appserver port [127.0.0.1:8065]: open Checking kvstore port [8191]: open Checking configuration... Done. Checking critical directories... Done Checking indexes... Validated: _audit _internal _introspection _metrics _metrics_rollup _telemetry _thefishbucket history main summary Done Checking filesystem compatibility... Done Checking conf files for problems... Done Checking default conf files for edits... Validating installed files against hashes from '/opt/splunk/splunk-8.2.1-ddff1c41e5cf-linux-2.6-x86_64-manifest' All installed files intact. Done All preliminary checks passed. Starting splunk server daemon (splunkd)... Done Waiting for web server at http://127.0.0.1:8000 to be available........ Done If you get stuck, we're here to help. Look for answers here: http://docs.splunk.com The Splunk web interface is at http://splunk:8000

Then I have waited 10 minutes and checked the result.

The number of the records is still increasing and the size of the index too.

The size of the index based on the query above is 1622.9306640625.

Re: limit the data size in Docker with maxTotalDataSizeMB does not work

codebuilder — Tue, 31 Aug 2021 15:04:56 GMT

Ok that looks good.

The other part then is that your search is returning a cumulative size of all indexes.
The config change you made affects only the "main" index.

Splunk indexes it's own logs to the _internaldb index, and that can grow quite large.
You would need to set maxTotalDataSizeMB globally using the [default] stanza, or set it for each index, including internal and any others.

Or limit your search to only "main" if that's what you're really curious about.
An easier method for that is:

# For all non-internal indexes | tstats count where index=* by index # For Splunk internal indexes | tstats count where index=_*

Re: limit the data size in Docker with maxTotalDataSizeMB does not work

codebuilder — Tue, 31 Aug 2021 15:13:07 GMT

Also just noticed, it may just be a copy/paste error here but you have a typo in your config:

rozenTimePeriodInSecs=300

Should be frozenTimePeriodInSecs=300
That could possibly cause the entire stanza or config to be ignored.

Re: limit the data size in Docker with maxTotalDataSizeMB does not work

somoarn — Wed, 01 Sep 2021 10:15:18 GMT

I have changed the configuration based on your recommendation.

This is the current setting I use:

$ cat /opt/splunk/etc/system/local/indexes.conf [default] maxTotalDataSizeMB=3 [main] maxTotalDataSizeMB=1 frozenTimePeriodInSecs=300 disabled=false

I also modified my queries a little bit:

Hopefully this shows the size in MB: index=_* source=* type=Usage idx=* | eval MB=b/1024/1024 | stats sum(MB) by st
result: 7.520857810974121, increasing

Number of records:

sourcetype=ADMIN_SERVER index="main" | stats count as Records
result: 1781, increasing

Size on disk:

| dbinspect index="main" | eval MB=sizeOnDiskMB| stats sum(MB)
result: 1.84375, increasing

According to my last query, the size of the main index that I use is 1.8MB which is bigger than 1MB that I set in the conf file.

I also checked the Settings > Monitoring Console > Index and volumes. Here I have found interesting reports.

As I can see all disk-related data is shown in GB, and I think that my feeding speed and size are not enough big to see changes in the charts. Maybe I need to change my "feeder" to inject more data into the logfile because my small echo "xxx" >> $logfile command does not send too many bytes to the file that I am monitoring.

But the most interesting thing that I see in the Index Detail: Instance report is this:

Retention policies

Re: limit the data size in Docker with maxTotalDataSizeMB does not work

somoarn — Wed, 01 Sep 2021 15:05:26 GMT

The following config works fine:

$ cat /opt/splunk/etc/system/local/indexes.conf [default] maxTotalDataSizeMB=5 maxDataSize=auto_high_volume [main] maxTotalDataSizeMB=3 frozenTimePeriodInSecs=300 maxDataSize=2 maxMemMB=1 maxHotBuckets=3 disabled=false

Re: limit the data size in Docker with maxTotalDataSizeMB does not work

codebuilder — Wed, 01 Sep 2021 17:16:26 GMT

Hey @somoarn I'm glad to hear we got this resolved for you. Even the slightest typo in a Splunk config can cause some unexpected behavior.

Configuring data retention, archiving, bucket rotation, etc. can become very complex. There are multiple layers of parameter settings and precedence rules that come into play.

One issue in your case was using "main", which is a pre-configured, Splunk index. Because you were setting only a few of the index parameters, you inherited the others from the Splunk configuration. Those settings combined with yours were preventing the bucket rotation to frozen/deleted that you were intending. But looks like you did a great job finding the right config combination that worked for you.

A couple of related notes worth mentioning...

From your original post it looked like the data you were creating for testing didn't include a timestamp. In that case you would need to have DATETIME_CONFIG = CURRENT defined in props.conf for you sourcetype. You may have it there already, but without it that can cause issues with aging out data as well.

Also, be very careful when you create a [default] stanza in /opt/splunk/etc/system/local/indexes.conf. Any parameter changes added there will be applied globally and affect every index in your environment. I know you're just testing on a container but its worth mentioning 😀

maxTotalDataSizeMB	1
frozenTimePeriodInSecs	300
homePath.maxDataSizeMB	0
coldPath.maxDataSizeMB	0