All Apps and Add-ons

New 7.2 feature will not work: journalCompression=zst

woodcock
Esteemed Legend

I am implementing Revealing the Magic on Splunk v7.2.4 from here:
https://static.rainfocus.com/splunk/splunkconf18/sess/15230307008970013eU6/finalPDF/FN1303_Revealing...

I cannot get the ZST compression working. First of all, I notice that they wrote index.conf on page 16 when I assume that they meant indexes.conf.

I created an app just for this, which I assume the authors did, too. In this app, I have tried:
1: Using journalCompression=zst all by itself to override the default value, but this did not work. 2: Using a stanza header for each index ( e.g. [_audit] ), each with that same journalCompression=zst line beneath it, but this did not work either!

For the latter, if I btool like this:
/opt/splunk/bin/splunk btool indexes list | egrep "journalCompression|zst|^["

Then I get this (which indicates it is OK):

[_audit]
journalCompression = zst
[_internal]
journalCompression = zst
[_introspection]
journalCompression = zst
[_telemetry]
journalCompression = zst
[_thefishbucket]
journalCompression = zst
[car_data]
journalCompression = zst
[cim_modactions]
journalCompression = zst
[default]
journalCompression = zst
[firedalerts]
journalCompression = zst
[history]
journalCompression = zst
[main]
journalCompression = zst
[os]
journalCompression = zst
[power_of_spl]
journalCompression = zst
[provider-family:hadoop]
[splexamples]
journalCompression = zst
[splexamples_downloadcount]
journalCompression = zst
[splexamples_mysummary]
journalCompression = zst
[splunklogger]
journalCompression = zst
[summary]
journalCompression = zst
[volume:_splunk_summaries]
journalCompression = zst
[whois]
journalCompression = zst

But after restart, when I run this:

find /opt/splunk/var/lib/splunk -name "*.zst"

It returns nothing, so the feature is clearly not active.
Not surprisingly, running this returns nothing:

/opt/splunk/bin/splunk btool check

On another 3-node Index cluster, I actually DO get errors trying to apply the bundle:
( /opt/splunk/bin/splunk show cluster-bundle-status 😞

master
         cluster_status=None
         active_bundle
                checksum=6BC53BF8B9FA9F10A38818E85CA2226C
                timestamp=1548996573 (in localtime=Thu Jan 31 23:49:33 2019)
         latest_bundle
                checksum=6BC53BF8B9FA9F10A38818E85CA2226C
                timestamp=1548996573 (in localtime=Thu Jan 31 23:49:33 2019)
         last_validated_bundle
                checksum=143308AF52A5F9606F4C60557CA30794
                last_validation_succeeded=0
                timestamp=1550442117 (in localtime=Sun Feb 17 17:21:57 2019)
         invalid_bundle
                checksum=143308AF52A5F9606F4C60557CA30794
                timestamp=1550442117 (in localtime=Sun Feb 17 17:21:57 2019)
                bundle_path=/opt/splunk/var/run/splunk/cluster/remote-bundle/d48fa52e996bba0be686541559e3ea2b-1550442117.bundle

<bundle_validation_errors on master>        
         last_check_restart_bundle
                last_check_restart_result=restart not required
                checksum=
                timestamp=0 (in localtime=Wed Dec 31 19:00:00 1969)

<bundle_validation_errors on peer>
[Critical]              stanza=_audit parameter=journalCompression Value supplied='zst' is illegal; default='gzip'
[Critical]              stanza=_internal parameter=journalCompression Value supplied='zst' is illegal; default='gzip'

...    

 aze-spl-idx01   A43CE47D-0B1B-4697-A1F2-6B2B1A1977E0    site1
         active_bundle=6BC53BF8B9FA9F10A38818E85CA2226C
         latest_bundle=6BC53BF8B9FA9F10A38818E85CA2226C
         last_validated_bundle=143308AF52A5F9606F4C60557CA30794
         last_bundle_validation_status=failure
         restart_required_apply_bundle=0
         status=Up

 aze-spl-idx02   B6DD0A86-368E-4BC3-BF1F-43B9BF0F3504    site1
         active_bundle=6BC53BF8B9FA9F10A38818E85CA2226C
         latest_bundle=6BC53BF8B9FA9F10A38818E85CA2226C
         last_validated_bundle=143308AF52A5F9606F4C60557CA30794
         last_bundle_validation_status=failure
         restart_required_apply_bundle=0
         status=Up

 aze-spl-idx03   EEBD9627-49E6-4C7B-B843-FC98BC9D5223    site1
         active_bundle=6BC53BF8B9FA9F10A38818E85CA2226C
         latest_bundle=6BC53BF8B9FA9F10A38818E85CA2226C
         last_validated_bundle=143308AF52A5F9606F4C60557CA30794
         last_bundle_validation_status=failure
         restart_required_apply_bundle=0

If you have gotten this feature to work, please share what version of splunk and a minimal sample of the working file.

0 Karma
1 Solution

spayneort
Contributor

I used the following all by itself in indexes.conf and it worked for me:

journalCompression = zstd

version 7.2.1

View solution in original post

0 Karma

spayneort
Contributor

I used the following all by itself in indexes.conf and it worked for me:

journalCompression = zstd

version 7.2.1

View solution in original post

0 Karma

woodcock
Esteemed Legend

I cannot believe it but that is it. THANK YOU SO MUCH!!!!!!!

0 Karma

gjanders
SplunkTrust
SplunkTrust

journalCompression = zst is invalid, the correct spelling in the original post is "zstd"

Alerts for Splunk Admins https://splunkbase.splunk.com/app/3796/
Version Control for Splunk https://splunkbase.splunk.com/app/4355/
0 Karma

woodcock
Esteemed Legend

I should have checked the documentation, which is correct.

0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!