Performance impact on uploading "X days ago" data

[robertlynch2020]

Hi

We have a situation where we can upload "live" or data from "X days ago". (They go into different indexes)
We have noticed that when we upload the X day old data we get the following messages.

• 01-14-2020 11:02:06.981 +0100 INFO HotBucketRoller - finished moving hot to warm bid=_internal~4425~DD9E7122-0692-45B5-AA4C-0500D72BC7A9 idx=_internal from=hot_v1_4425 to=db_1578459814_1578333458_4425 size=49152 caller=lru maxHotBuckets=3, count=13 hot buckets,evicting_count=10 LRU hots

We then have performance issues.
I think we are making a mess of the caches (Hot warm buckets)... - As the data is going into different caches can we separate the warm to hot per index (So "live" to "X days ago" )?

Below is an example of an upload, as you can see 1 Million events goes in as 4 days ago (Functionally this is correct). But we get slowness.

[richgalloway]

This is to be expected. The old data doesn't fit into an existing hot bucket because the events precede everything in those buckets. Therefore, Splunk must create new hot buckets for the events. This may cause it to exceed the number of hot buckets allowed so some will be rolled to warm. Once the old data is ingested, those hot buckets likely will be too old and will immediately get rolled to warm. Rolling from hot to warm is pretty simple, but if you have 1 million events then the cumulative effect of all those buckets may be noticeable.

Consider revising your indexes.conf settings for the old data. For instance, increasing maxHotBuckets may help.

[robertlynch2020]

Hi

Thanks for the replay.
So when when i look inside indexs.conf (Default). I see that prop in 2 locations.

################################################################################
# index definitions
################################################################################

[main]
 maxHotBuckets = 10

and

index specific defaults
maxHotBuckets = 3

So i am not sure what is the one i should increase?

[richgalloway]

Increase the one specific to the index having the problem. If there is not a setting for that index, add it.

[robertlynch2020]

HI

Sorry for the delay on gettign back on this and thanks for the repay.

I am now getting this error for _internal. I am on a single install, so i am assuming i can also applay what you have said above?

01-28-2020 03:02:43.599 +0100 INFO HotBucketRoller - finished moving hot to warm bid=_internal~4670~DD9E7122-0692-45B5-AA4C-0500D72BC7A9 idx=_internal from=hot_v1_4670 to=db_1580006625_1574337362_4670 size=238698496 caller=lru maxHotBuckets=3, count=4 hot buckets,evicting_count=1 LRU hots