Deployment Architecture

How to fix the searches across the reduced buckets issue for a particular index ?

Motivator

Hi All, Currently we are facing an issue while performing a search against a particular index and found it was due to the searches across the reduced buckets and getting this error in Job and it is running very slow.

Error message : Search on most recent data has completed. Expect slower search speeds as we search the reduced buckets.

query : earliest=-24h index=windows sourcetype="WinEventLog:Security" EventCode=4735

However the tsidx should not be reduced on a search that is in the past 24 hours. Our tsidx reduction is set for 90 days.

stanza details
[default]
frozenTimePeriodInSecs = 31536000
enableTsidxReduction = true
timePeriodInSecBeforeTsidxReduction = 7884000

Kindly let me know how to troubleshoot this issue.
thanks in advance.

0 Karma

Motivator

Hi Iguinn, thanks for your effort, when I ran the btool check on the indexer instance, I have got the below output but not sure whether this is related to the issue. Kindly guide me if it related to this issue.

Note: The Admin-all_indexers app contains all the index config details

Invalid stanza [_blocksignature] in /opt/splunk/etc/apps/ADMIN-all_indexers/local/indexes.conf, line 197. The block-signing feature is no longer available in Splunk. Please remove stanza=[_blocksignature] from the indexes.conf. For further details, please refer to the related topic in the latest version of 'Securing Splunk' manual on docs.splunk.com.
Invalid key in stanza [search] in /opt/splunk/etc/system/local/limits.conf, line 166: max_results_raw_size (value: 100000000).
Invalid key in stanza [LDAP] in /opt/splunk/etc/apps/ADMIN-all_indexers/default/authentication.conf, line 6: charses (value: utf8).
Invalid key in stanza [search] in /opt/splunk/etc/system/local/limits.conf, line 243: multi_threaded_setup (value: false).
Invalid key in stanza [scheduler] in /opt/splunk/etc/system/local/limits.conf, line 409: peristance_period (value: 30).
Invalid key in stanza [tscollect] in /opt/splunk/etc/system/local/limits.conf, line 467: tsidx_init_file_goal_mb (value: 500).

0 Karma

Motivator

Hi lguinn, can you guide me whether the above output provide in the comment is something to do with the issue. Or is there any other troubleshooting method which we need to follow to fix it.

thanks in advance.

0 Karma

Legend

All of those messages show configuration errors, which you should fix immediately.

According to the documentation (indexes.conf.spec)

enableTsidxReduction = true|false
tsidxReductionCheckPeriodInSec = <positive integer>

should be set for EACH INDEX, not globally. So you also need to fix the indexes.conf file in Admin-all_indexers app and redeploy it.
Sadly, once a bucket is reduced, changing the settings in indexes.conf will not expand it back. But at least fixing the indexes.conf should help prevent buckets from being compressed too soon in the future.

I suggest that you file a Support ticket if this does not help.

0 Karma

Legend

You should file a Support ticket. Buckets with data less than 24 hours old should not be reduced - if your configuration is correct.

Are you sure that you can put the timePeriodInSecBeforeTsidxReduction = 7884000 in the default stanza? Perhaps it needs to be in the stanza for each index individually.

Also, have you run a btool check to make sure that there are no syntax errors in your configuration files? On linux:

./splunk btool check

Finally, searches over reduced buckets will be very slow. That is expected. And 90 days is actually 7776000

0 Karma