_internal hot to warm buckets causing issue

[robertlynch2020]

Hi

I have the following error and i am not sure how to increase the _internal buckets

Root Cause(s):
The percentage of small of buckets created (100) over the last hour is very high and exceeded the red thresholds (90) for index=_internal, and possibly more indexes, on this indexer
Last 50 related messages:
03-10-2020 12:34:23.745 +0100 INFO HotBucketRoller - finished moving hot to warm bid=_internal~4968~DD9E7122-0692-45B5-AA4C-0500D72BC7A9 idx=_internal from=hot_v1_4968 to=db_1547726203_1547726203_4968 size=40960 caller=lru maxHotBuckets=3, count=4 hot buckets,evicting_count=1 LRU hots
03-10-2020 11:53:10.742 +0100 INFO HotBucketRoller - finished moving hot to warm bid=_internal~4967~DD9E7122-0692-45B5-AA4C-0500D72BC7A9 idx=_internal from=hot_v1_4967 to=db_1582194881_1582194881_4967 size=45056 caller=lru maxHotBuckets=3, count=4 hot buckets,evicting_count=1 LRU hots
03-10-2020 03:56:16.392 +0100 INFO HotBucketRoller - finished moving hot to warm bid=_internal~4966~DD9E7122-0692-45B5-AA4C-0500D72BC7A9 idx=_internal from=hot_v1_4966 to=db_1582194881_1582194881_4966 size=40960 caller=lru maxHotBuckets=3, count=4 hot buckets,evicting_count=1 LRU hots
03-10-2020 01:00:25.190 +0100 INFO HotBucketRoller - finished moving hot to warm bid=_internal~4965~DD9E7122-0692-45B5-AA4C-0500D72BC7A9 idx=_internal from=hot_v1_4965 to=db_1547726203_1547726203_4965 size=40960 caller=lru maxHotBuckets=3, count=4 hot buckets,evicting_count=1 LRU hots

[xavierashe]

Look at your indexes.conf. What is your maxDataSize and maxHotBuckets for the _internal index?

[robertlynch2020]

Hi

I don't have an indexes.conf defined in /hp737srv2/apps/splunk/etc/system/local

In default it is below - should i create the file and perhaps increase?
[_internal]
homePath = $SPLUNK_DB/_internaldb/db
coldPath = $SPLUNK_DB/_internaldb/colddb
thawedPath = $SPLUNK_DB/_internaldb/thaweddb
tstatsHomePath = volume:_splunk_summaries/_internaldb/datamodel_summary
maxDataSize = 1000
maxHotSpanSecs = 432000
frozenTimePeriodInSecs = 2592000

[harsmarvania57 · Accepted solution]

Based on message it looks like it is reporting wrongly, in your screenshot only 4 buckets moved from hot to warm. Can you please check how many hot bucket created for _internal index in last one hour using below query ?

index=_internal host=<Your INDEXER> source="/opt/splunk/var/log/splunk/splunkd.log" component=IndexWriter
| stats count by idx

[robertlynch2020]

Hi

Thanks for the replay.

when i run below for the last 1 hour i get 0 results, but there is still a red ball in the
index=_internal host=hp737srv component=IndexWriter source="/hp737srv2/apps/splunk/var/log/splunk/splunkd.log" | stats count by idx

when i run for last 7 hours i get
idx count
_internal 58

So it all looks low, but i still have a red message

[harsmarvania57]

And what about if you run for last 24 hours ?

[robertlynch2020]

So sorry my original comment was incorrect.

index=_internal component=IndexWriter source=*splunkd.log | stats count by idx

Last 60 minutes = 0
Last 24 hours = 6
Last 7 days = 57

This is all index for last 24 hours
idx count
_audit 1
_internal 6
_telemetry 1
mlc_live 4
mlc_log_drop 3

To me these numbers are not high, so i am not sure why i am getting the red alert.
+ When i click on it only displays 4 - it says last 50 related messages, but it gives only 5

 Buckets
Root Cause(s):
The percentage of small of buckets created (100) over the last hour is very high and exceeded the red thresholds (90) for index=_internal, and possibly more indexes, on this indexer
Last 50 related messages:
03-10-2020 16:10:36.977 +0100 INFO HotBucketRoller - finished moving hot to warm bid=mlc_live~8118~DD9E7122-0692-45B5-AA4C-0500D72BC7A9 idx=mlc_live from=hot_v1_8118 to=db_1583533443_1582047188_8118 size=931500032 caller=size_exceeded _maxHotBucketSize=786432000 (750MB), bucketSize=1036042240 (988MB)
03-10-2020 12:34:23.745 +0100 INFO HotBucketRoller - finished moving hot to warm bid=_internal~4968~DD9E7122-0692-45B5-AA4C-0500D72BC7A9 idx=_internal from=hot_v1_4968 to=db_1547726203_1547726203_4968 size=40960 caller=lru maxHotBuckets=3, count=4 hot buckets,evicting_count=1 LRU hots
03-10-2020 11:53:10.742 +0100 INFO HotBucketRoller - finished moving hot to warm bid=_internal~4967~DD9E7122-0692-45B5-AA4C-0500D72BC7A9 idx=_internal from=hot_v1_4967 to=db_1582194881_1582194881_4967 size=45056 caller=lru maxHotBuckets=3, count=4 hot buckets,evicting_count=1 LRU hots
03-10-2020 03:56:16.392 +0100 INFO HotBucketRoller - finished moving hot to warm bid=_internal~4966~DD9E7122-0692-45B5-AA4C-0500D72BC7A9 idx=_internal from=hot_v1_4966 to=db_1582194881_1582194881_4966 size=40960 caller=lru maxHotBuckets=3, count=4 hot buckets,evicting_count=1 LRU hots
03-10-2020 01:00:25.190 +0100 INFO HotBucketRoller - finished moving hot to warm bid=_internal~4965~DD9E7122-0692-45B5-AA4C-0500D72BC7A9 idx=_internal from=hot_v1_4965 to=db_1547726203_1547726203_4965 size=40960 caller=lru maxHotBuckets=3, count=4 hot buckets,evicting_count=1 LRU hots

[harsmarvania57]

Yes so it looks like splunk is reporting wrong number, I can see same issue on 7.2.7

[robertlynch2020]

Agreed, if you post it as an answer i will accept it.

[harsmarvania57]

You can look at number of buckets moved from hot to warm using below query

index=_internal host=YOUR_INDEXER source="/opt/splunk/var/log/splunk/splunkd.log" component=HotBucketRoller

| stats count by idx

[robertlynch2020]

hi

Thanks for your help, this was in the last 24 hours

idx count
_internal 3
mlc_live 1
mxtiming_live 7

[robertlynch2020]

We are on 7.2.6, so we think this is a bug?

[Stu]

I'm having very similar issue on 8.2.2.1 and the only thing I can think of is adding new stanzas to index=_internal which is not a good idea

my results are:

idx count
_internal 2
msad 2
win-security 3