limit the data size in Docker with maxTotalDataSizeMB does not work

[somoarn]

I am learning Splunk and I have built the following test environment in Docker:

• Splunk server running in a container, using the official docker image:  image: splunk/splunk:8.2
• I have another docker container, call it client where I installed the forwarder and then I added a file to monitor with the $SPLUNK_HOME/bin/splunk add monitor $MY_LOGFILE -index main -sourcetype mylog command.

Everything works fine. If I append $MY_LOGFILE in the client docker container with 

echo "hello" >> $MY_LOGFILE

command then I can see the new line in the Splunk web console.

Now I am appending/feeding my log file with an endless bash counter-up loop and I can see everything in the Splunk web console. Great.

My question:

I would like to delete old records from Splunk to save disk space, so I followed the documentation and I did this:

sudo vi /opt/splunk/etc/system/local/indexes.conf

with this content

[main]
maxTotalDataSizeMB=1
rozenTimePeriodInSecs=300
disabled=false

As I know this allows Splunk to automatically delete old data when my index hits the 1MB size.  After I have created this new config file, I restarted the Splunk Docker container (and Splunk as well manually).

But actually, nothing happens. It seems that this setting is not considered, and I see the increasing number of records in the index and index size is also increasing without limitation in Splunk.

I use the following commands to check index size:

• sourcetype=mylog | stats count as Records
• index=_internal source=* type=Usage idx=* | eval SIZE=b/1024 | stats sum(SIZE) by st, result: 30756.775390625

But when I stop Splunk then I am able to clean up the index with this command:

splunk stop
splunk clean eventdata  
splunk start

But I have a scenario where I need to limit the size of the index and the disk usage that is used by Splunk index "realtime", without stop and start. 

What I am missing here?

Thx

[somoarn]

The following config works fine:

$ cat /opt/splunk/etc/system/local/indexes.conf
[default]
maxTotalDataSizeMB=5
maxDataSize=auto_high_volume

[main]
maxTotalDataSizeMB=3
frozenTimePeriodInSecs=300
maxDataSize=2
maxMemMB=1
maxHotBuckets=3
disabled=false

[codebuilder]

Hey @somoarn I'm glad to hear we got this resolved for you. Even the slightest typo in a Splunk config can cause some unexpected behavior.

Configuring data retention, archiving, bucket rotation, etc. can become very complex. There are multiple layers of parameter settings and precedence rules that come into play.

One issue in your case was using "main", which is a pre-configured, Splunk index. Because you were setting only a few of the index parameters, you inherited the others from the Splunk configuration. Those settings combined with yours were preventing the bucket rotation to frozen/deleted that you were intending. But looks like you did a great job finding the right config combination that worked for you.

A couple of related notes worth mentioning...

From your original post it looked like the data you were creating for testing didn't include a timestamp. In that case you would need to have DATETIME_CONFIG = CURRENT defined in props.conf for you sourcetype. You may have it there already, but without it that can cause issues with aging out data as well.

Also, be very careful when you create a [default] stanza in /opt/splunk/etc/system/local/indexes.conf. Any parameter changes added there will be applied globally and affect every index in your environment. I know you're just testing on a container but its worth mentioning 😀

[somoarn]

I have changed the configuration based on your recommendation.

This is the current setting I use:

$ cat /opt/splunk/etc/system/local/indexes.conf
[default]
maxTotalDataSizeMB=3

[main]
maxTotalDataSizeMB=1
frozenTimePeriodInSecs=300
disabled=false

I also modified my queries a little bit:

• Hopefully this shows the size in MB: index=_* source=* type=Usage idx=* | eval MB=b/1024/1024 | stats sum(MB) by st
• result: 7.520857810974121, increasing

Number of records:

• sourcetype=ADMIN_SERVER index="main" | stats count as Records
• result: 1781, increasing

Size on disk:

• | dbinspect index="main" | eval MB=sizeOnDiskMB| stats sum(MB)
• result: 1.84375, increasing

According to my last query, the size of the main index that I use is 1.8MB which is bigger than 1MB that I set in the conf file.

I also checked the Settings > Monitoring Console > Index and volumes. Here I have found interesting reports.

As I can see all disk-related data is shown in GB, and I think that my feeding speed and size are not enough big to see changes in the charts. Maybe I need to change my "feeder" to inject more data into the logfile because my small echo "xxx" >> $logfile  command does not send too many bytes to the file that I am monitoring.

But the most interesting thing that I see in the Index Detail: Instance report is this:

Retention policies

maxTotalDataSizeMB	1
frozenTimePeriodInSecs	300
homePath.maxDataSizeMB	0
coldPath.maxDataSizeMB	0

It seems that my settings is applied.

I see in the report that the main index I use received 2465 events and it has only 1 bucket.

So maybe I need to wait a little bit to have 2 buckets and then the increasing of records and disk usage will stop.

 I so appreciate your help, your advice helped a lot.

If you could suggest to me something else that I could check/configure that would be great.

In the meantime, I will change my feeding bash script to send more data to Splunk.

thx a lot

[somoarn]

Thx for the response. This is a good idea. 

I have dropped my containers and started again from the scratch.

$ docker exec -u splunk -it splunk /bin/bash
vi /opt/splunk/etc/system/local/indexes.conf

[main]
maxTotalDataSizeMB=1
frozenTimePeriodInSecs=300
disabled=false

ls -all /opt/splunk/etc/system/local/indexes.conf
-rw-rw-r-- 1 splunk splunk 71 Aug 31 14:58 /opt/splunk/etc/system/local/indexes.conf

Then I restarted the Splunk this way:

$ cd /opt/splunk/bin/
$ ./splunk restart
Stopping splunkd...
Shutting down.  Please wait, as this may take a few minutes.
.....................................
Stopping splunk helpers...

Done.

Splunk> Needle. Haystack. Found.

Checking prerequisites...
	Checking http port [8000]: open
	Checking mgmt port [8089]: open
	Checking appserver port [127.0.0.1:8065]: open
	Checking kvstore port [8191]: open
	Checking configuration... Done.
	Checking critical directories...	Done
	Checking indexes...
		Validated: _audit _internal _introspection _metrics _metrics_rollup _telemetry _thefishbucket history main summary
	Done
	Checking filesystem compatibility...  Done
	Checking conf files for problems...
	Done
	Checking default conf files for edits...
	Validating installed files against hashes from '/opt/splunk/splunk-8.2.1-ddff1c41e5cf-linux-2.6-x86_64-manifest'
	All installed files intact.
	Done
All preliminary checks passed.

Starting splunk server daemon (splunkd)...  
Done


Waiting for web server at http://127.0.0.1:8000 to be available........ Done


If you get stuck, we're here to help.  
Look for answers here: http://docs.splunk.com

The Splunk web interface is at http://splunk:8000

Then I have waited 10 minutes and checked the result.

The number of the records is still increasing and the size of the index too.

The size of the index based on the query above is 1622.9306640625.

[codebuilder · Accepted solution]

Ok that looks good.

The other part then is that your search is returning a cumulative size of all indexes.
The config change you made affects only the "main" index.

Splunk indexes it's own logs to the _internaldb index, and that can grow quite large.
You would need to set maxTotalDataSizeMB globally using the [default] stanza, or set it for each index, including internal and any others.

Or limit your search to only "main" if that's what you're really curious about.
An easier method for that is:

# For all non-internal indexes
| tstats count where index=* by index

# For Splunk internal indexes
| tstats count where index=_*

[codebuilder]

Also just noticed, it may just be a copy/paste error here but you have a typo in your config:

rozenTimePeriodInSecs=300
 
Should be frozenTimePeriodInSecs=300
That could possibly cause the entire stanza or config to be ignored.

[codebuilder]

This could be caused by several issues.
The first to check is the ownership of your indexes.conf. I noticed you used sudo to edit it, and therefore it will be owned by root. Splunk runs as the "splunk" user by default. Because of that it will ignore the file you put into local because it doesn't have permission to read it. Unless you are explicitly running Splunk as root. Change the ownership of the file and all directories above.

Next would be restarting the container itself. Unless you have persistent storage configured, the container will pick a new directory for storage each time it is restarted.

There are other causes, but those are where I would start, and lean towards the ownership of indexes.conf.