All Apps and Add-ons

New 7.2 feature will not work: journalCompression=zst

woodcock
Esteemed Legend

I am implementing Revealing the Magic on Splunk v7.2.4 from here:
https://static.rainfocus.com/splunk/splunkconf18/sess/15230307008970013eU6/finalPDF/FN1303_Revealing...

I cannot get the ZST compression working. First of all, I notice that they wrote index.conf on page 16 when I assume that they meant indexes.conf.

I created an app just for this, which I assume the authors did, too. In this app, I have tried:
1: Using journalCompression=zst all by itself to override the default value, but this did not work. 2: Using a stanza header for each index ( e.g. [_audit] ), each with that same journalCompression=zst line beneath it, but this did not work either!

For the latter, if I btool like this:
/opt/splunk/bin/splunk btool indexes list | egrep "journalCompression|zst|^["

Then I get this (which indicates it is OK):

[_audit]
journalCompression = zst
[_internal]
journalCompression = zst
[_introspection]
journalCompression = zst
[_telemetry]
journalCompression = zst
[_thefishbucket]
journalCompression = zst
[car_data]
journalCompression = zst
[cim_modactions]
journalCompression = zst
[default]
journalCompression = zst
[firedalerts]
journalCompression = zst
[history]
journalCompression = zst
[main]
journalCompression = zst
[os]
journalCompression = zst
[power_of_spl]
journalCompression = zst
[provider-family:hadoop]
[splexamples]
journalCompression = zst
[splexamples_downloadcount]
journalCompression = zst
[splexamples_mysummary]
journalCompression = zst
[splunklogger]
journalCompression = zst
[summary]
journalCompression = zst
[volume:_splunk_summaries]
journalCompression = zst
[whois]
journalCompression = zst

But after restart, when I run this:

find /opt/splunk/var/lib/splunk -name "*.zst"

It returns nothing, so the feature is clearly not active.
Not surprisingly, running this returns nothing:

/opt/splunk/bin/splunk btool check

On another 3-node Index cluster, I actually DO get errors trying to apply the bundle:
( /opt/splunk/bin/splunk show cluster-bundle-status 😞

master
         cluster_status=None
         active_bundle
                checksum=6BC53BF8B9FA9F10A38818E85CA2226C
                timestamp=1548996573 (in localtime=Thu Jan 31 23:49:33 2019)
         latest_bundle
                checksum=6BC53BF8B9FA9F10A38818E85CA2226C
                timestamp=1548996573 (in localtime=Thu Jan 31 23:49:33 2019)
         last_validated_bundle
                checksum=143308AF52A5F9606F4C60557CA30794
                last_validation_succeeded=0
                timestamp=1550442117 (in localtime=Sun Feb 17 17:21:57 2019)
         invalid_bundle
                checksum=143308AF52A5F9606F4C60557CA30794
                timestamp=1550442117 (in localtime=Sun Feb 17 17:21:57 2019)
                bundle_path=/opt/splunk/var/run/splunk/cluster/remote-bundle/d48fa52e996bba0be686541559e3ea2b-1550442117.bundle

<bundle_validation_errors on master>        
         last_check_restart_bundle
                last_check_restart_result=restart not required
                checksum=
                timestamp=0 (in localtime=Wed Dec 31 19:00:00 1969)

<bundle_validation_errors on peer>
[Critical]              stanza=_audit parameter=journalCompression Value supplied='zst' is illegal; default='gzip'
[Critical]              stanza=_internal parameter=journalCompression Value supplied='zst' is illegal; default='gzip'

...    

 aze-spl-idx01   A43CE47D-0B1B-4697-A1F2-6B2B1A1977E0    site1
         active_bundle=6BC53BF8B9FA9F10A38818E85CA2226C
         latest_bundle=6BC53BF8B9FA9F10A38818E85CA2226C
         last_validated_bundle=143308AF52A5F9606F4C60557CA30794
         last_bundle_validation_status=failure
         restart_required_apply_bundle=0
         status=Up

 aze-spl-idx02   B6DD0A86-368E-4BC3-BF1F-43B9BF0F3504    site1
         active_bundle=6BC53BF8B9FA9F10A38818E85CA2226C
         latest_bundle=6BC53BF8B9FA9F10A38818E85CA2226C
         last_validated_bundle=143308AF52A5F9606F4C60557CA30794
         last_bundle_validation_status=failure
         restart_required_apply_bundle=0
         status=Up

 aze-spl-idx03   EEBD9627-49E6-4C7B-B843-FC98BC9D5223    site1
         active_bundle=6BC53BF8B9FA9F10A38818E85CA2226C
         latest_bundle=6BC53BF8B9FA9F10A38818E85CA2226C
         last_validated_bundle=143308AF52A5F9606F4C60557CA30794
         last_bundle_validation_status=failure
         restart_required_apply_bundle=0

If you have gotten this feature to work, please share what version of splunk and a minimal sample of the working file.

0 Karma
1 Solution

spayneort
Contributor

I used the following all by itself in indexes.conf and it worked for me:

journalCompression = zstd

version 7.2.1

View solution in original post

0 Karma

spayneort
Contributor

I used the following all by itself in indexes.conf and it worked for me:

journalCompression = zstd

version 7.2.1

0 Karma

woodcock
Esteemed Legend

I cannot believe it but that is it. THANK YOU SO MUCH!!!!!!!

0 Karma

gjanders
SplunkTrust
SplunkTrust

journalCompression = zst is invalid, the correct spelling in the original post is "zstd"

0 Karma

woodcock
Esteemed Legend

I should have checked the documentation, which is correct.

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...