Other Usage

_internal hot to warm buckets causing issue

robertlynch2020
Motivator

Hi

I have the following error and i am not sure how to increase the _internal buckets

Root Cause(s):
The percentage of small of buckets created (100) over the last hour is very high and exceeded the red thresholds (90) for index=_internal, and possibly more indexes, on this indexer
Last 50 related messages:
03-10-2020 12:34:23.745 +0100 INFO HotBucketRoller - finished moving hot to warm bid=_internal~4968~DD9E7122-0692-45B5-AA4C-0500D72BC7A9 idx=_internal from=hot_v1_4968 to=db_1547726203_1547726203_4968 size=40960 caller=lru maxHotBuckets=3, count=4 hot buckets,evicting_count=1 LRU hots
03-10-2020 11:53:10.742 +0100 INFO HotBucketRoller - finished moving hot to warm bid=_internal~4967~DD9E7122-0692-45B5-AA4C-0500D72BC7A9 idx=_internal from=hot_v1_4967 to=db_1582194881_1582194881_4967 size=45056 caller=lru maxHotBuckets=3, count=4 hot buckets,evicting_count=1 LRU hots
03-10-2020 03:56:16.392 +0100 INFO HotBucketRoller - finished moving hot to warm bid=_internal~4966~DD9E7122-0692-45B5-AA4C-0500D72BC7A9 idx=_internal from=hot_v1_4966 to=db_1582194881_1582194881_4966 size=40960 caller=lru maxHotBuckets=3, count=4 hot buckets,evicting_count=1 LRU hots
03-10-2020 01:00:25.190 +0100 INFO HotBucketRoller - finished moving hot to warm bid=_internal~4965~DD9E7122-0692-45B5-AA4C-0500D72BC7A9 idx=_internal from=hot_v1_4965 to=db_1547726203_1547726203_4965 size=40960 caller=lru maxHotBuckets=3, count=4 hot buckets,evicting_count=1 LRU hots

alt text

Labels (1)
Tags (1)
0 Karma
1 Solution

harsmarvania57
SplunkTrust
SplunkTrust

Based on message it looks like it is reporting wrongly, in your screenshot only 4 buckets moved from hot to warm. Can you please check how many hot bucket created for _internal index in last one hour using below query ?

index=_internal host=<Your INDEXER> source="/opt/splunk/var/log/splunk/splunkd.log" component=IndexWriter
| stats count by idx

View solution in original post

0 Karma

xavierashe
Contributor

Look at your indexes.conf. What is your maxDataSize and maxHotBuckets for the _internal index?

0 Karma

robertlynch2020
Motivator

Hi

I don't have an indexes.conf defined in /hp737srv2/apps/splunk/etc/system/local

In default it is below - should i create the file and perhaps increase?
[_internal]
homePath = $SPLUNK_DB/_internaldb/db
coldPath = $SPLUNK_DB/_internaldb/colddb
thawedPath = $SPLUNK_DB/_internaldb/thaweddb
tstatsHomePath = volume:_splunk_summaries/_internaldb/datamodel_summary
maxDataSize = 1000
maxHotSpanSecs = 432000
frozenTimePeriodInSecs = 2592000

0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

Based on message it looks like it is reporting wrongly, in your screenshot only 4 buckets moved from hot to warm. Can you please check how many hot bucket created for _internal index in last one hour using below query ?

index=_internal host=<Your INDEXER> source="/opt/splunk/var/log/splunk/splunkd.log" component=IndexWriter
| stats count by idx
0 Karma

robertlynch2020
Motivator

Hi

Thanks for the replay.

when i run below for the last 1 hour i get 0 results, but there is still a red ball in the
index=_internal host=hp737srv component=IndexWriter source="/hp737srv2/apps/splunk/var/log/splunk/splunkd.log" | stats count by idx

when i run for last 7 hours i get
idx count
_internal 58

So it all looks low, but i still have a red message

0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

And what about if you run for last 24 hours ?

0 Karma

robertlynch2020
Motivator

So sorry my original comment was incorrect.

index=_internal component=IndexWriter source=*splunkd.log | stats count by idx

Last 60 minutes = 0
Last 24 hours = 6
Last 7 days = 57

This is all index for last 24 hours
idx count
_audit 1
_internal 6
_telemetry 1
mlc_live 4
mlc_log_drop 3

To me these numbers are not high, so i am not sure why i am getting the red alert.
+ When i click on it only displays 4 - it says last 50 related messages, but it gives only 5

 Buckets
Root Cause(s):
The percentage of small of buckets created (100) over the last hour is very high and exceeded the red thresholds (90) for index=_internal, and possibly more indexes, on this indexer
Last 50 related messages:
03-10-2020 16:10:36.977 +0100 INFO HotBucketRoller - finished moving hot to warm bid=mlc_live~8118~DD9E7122-0692-45B5-AA4C-0500D72BC7A9 idx=mlc_live from=hot_v1_8118 to=db_1583533443_1582047188_8118 size=931500032 caller=size_exceeded _maxHotBucketSize=786432000 (750MB), bucketSize=1036042240 (988MB)
03-10-2020 12:34:23.745 +0100 INFO HotBucketRoller - finished moving hot to warm bid=_internal~4968~DD9E7122-0692-45B5-AA4C-0500D72BC7A9 idx=_internal from=hot_v1_4968 to=db_1547726203_1547726203_4968 size=40960 caller=lru maxHotBuckets=3, count=4 hot buckets,evicting_count=1 LRU hots
03-10-2020 11:53:10.742 +0100 INFO HotBucketRoller - finished moving hot to warm bid=_internal~4967~DD9E7122-0692-45B5-AA4C-0500D72BC7A9 idx=_internal from=hot_v1_4967 to=db_1582194881_1582194881_4967 size=45056 caller=lru maxHotBuckets=3, count=4 hot buckets,evicting_count=1 LRU hots
03-10-2020 03:56:16.392 +0100 INFO HotBucketRoller - finished moving hot to warm bid=_internal~4966~DD9E7122-0692-45B5-AA4C-0500D72BC7A9 idx=_internal from=hot_v1_4966 to=db_1582194881_1582194881_4966 size=40960 caller=lru maxHotBuckets=3, count=4 hot buckets,evicting_count=1 LRU hots
03-10-2020 01:00:25.190 +0100 INFO HotBucketRoller - finished moving hot to warm bid=_internal~4965~DD9E7122-0692-45B5-AA4C-0500D72BC7A9 idx=_internal from=hot_v1_4965 to=db_1547726203_1547726203_4965 size=40960 caller=lru maxHotBuckets=3, count=4 hot buckets,evicting_count=1 LRU hots
0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

Yes so it looks like splunk is reporting wrong number, I can see same issue on 7.2.7

robertlynch2020
Motivator

Agreed, if you post it as an answer i will accept it.

0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

You can look at number of buckets moved from hot to warm using below query

index=_internal host=YOUR_INDEXER source="/opt/splunk/var/log/splunk/splunkd.log" component=HotBucketRoller

| stats count by idx

0 Karma

robertlynch2020
Motivator

hi

Thanks for your help, this was in the last 24 hours

idx count
_internal 3
mlc_live 1
mxtiming_live 7

0 Karma

robertlynch2020
Motivator

We are on 7.2.6, so we think this is a bug?

0 Karma

Stu
New Member

I'm having very similar issue on 8.2.2.1 and the only thing I can think of is adding new stanzas to index=_internal which is not a good idea

my results are:

idx count
_internal 2
msad 2
win-security 3

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...