Dashboards & Visualizations

Default size for _internal index

gcusello
SplunkTrust
SplunkTrust

Hi at all,
what't the default size of _internal index:

  • 5 GB
  • 10 GB
  • 20 GB
  • 30 GB

I'm working on Splunk from 8 years and I didn't find any special default size for this index (except the normal size of 500 GB of all indexes).
Anyone can say to me where can I find this dimension (if exists)?

Thank you.
Giuseppe

0 Karma
1 Solution

Sukisen1981
Champion

hi @gcusello Firstly, it doesn't matter at all whether you passed or failed, you were and will remain one of the most knowledgeable forum members, the kind of guy who makes others NOT to attempt answers for the questions that you have already answered, for they are always correct!
The question is tricky , I did try to make some sense of it from my local. I may (most probably will be) wrong.
alt text

As you say there is a max size of the entire index(and all indexes) in general that can go up to 500 GB. That is more like an upper limit perhaps?
Below that is given max size of hot/warm/cold bucket ~ 1 GB each. Now, if the buckets were created with the deafult max size of 1 gb each and assuming we include 5 buckets hot,warm,cold,frozen and thawed it would perhaps be 5 gb in total.
There is a separate path for thawed db and an optional path for frozen as well in the same settings screen shot.
If you refer this - https://docs.splunk.com/Documentation/Splunk/7.3.1/Indexer/HowSplunkstoresindexes#Bucket_names
and see the section 'how data ages'
A bucket moves through several states as it ages:

hot
warm
cold
frozen
thawed
As buckets age, they "roll" from one state to the next. When data is first indexed, it gets written to a hot bucket. Hot buckets are buckets that actively being written to. An index can have several hot buckets open at a time. Hot buckets are also searchable.

When certain conditions are met (for example, the hot bucket reaches a certain size or the indexer gets restarted), the hot bucket becomes a warm bucket ("rolls to warm"), and a new hot bucket is created in its place. The warm bucket is renamed but it remains in the same location as when it was a hot bucket. Warm buckets are searchable, but they are not actively written to. There can be a large number of warm buckets.

Once further conditions are met (for example, the index reaches some maximum number of warm buckets), the indexer begins to roll the warm buckets to cold, based on their age. It always selects the oldest warm bucket to roll to cold. Buckets continue to roll to cold as they age in this manner. Cold buckets reside in a different location from hot and warm buckets. You can configure the location so that cold buckets reside on cheaper storage.

Finally, after certain other time-based or size-based conditions are met, cold buckets roll to the frozen state, at which point they are deleted from the index, after being optionally archived.

If the frozen data has been archived, it can later be thawed. Data in thawed buckets is available for searches.

Settings in indexes.conf determine when a bucket moves from one state to the next.

They talk about 5 buckets and assuming that they already come with some default size(the max size per bucket) it could be 5 gb . answer#1.
It does also say that frozen buckets are deleted by default unless they are archived so I am not all sure!
I am just perhaps trying to force fit an answer given the options that you mentioned...
or could it be a typo in the question from the exam itself!!?
Sorry, if you had already considered these options and decided that they were incorrect.

View solution in original post

woodcock
Esteemed Legend

I agree that the test focuses too much on stuff that is not important to have memorized. Too much CLI syntax, too.

0 Karma

Sukisen1981
Champion

hi @gcusello Firstly, it doesn't matter at all whether you passed or failed, you were and will remain one of the most knowledgeable forum members, the kind of guy who makes others NOT to attempt answers for the questions that you have already answered, for they are always correct!
The question is tricky , I did try to make some sense of it from my local. I may (most probably will be) wrong.
alt text

As you say there is a max size of the entire index(and all indexes) in general that can go up to 500 GB. That is more like an upper limit perhaps?
Below that is given max size of hot/warm/cold bucket ~ 1 GB each. Now, if the buckets were created with the deafult max size of 1 gb each and assuming we include 5 buckets hot,warm,cold,frozen and thawed it would perhaps be 5 gb in total.
There is a separate path for thawed db and an optional path for frozen as well in the same settings screen shot.
If you refer this - https://docs.splunk.com/Documentation/Splunk/7.3.1/Indexer/HowSplunkstoresindexes#Bucket_names
and see the section 'how data ages'
A bucket moves through several states as it ages:

hot
warm
cold
frozen
thawed
As buckets age, they "roll" from one state to the next. When data is first indexed, it gets written to a hot bucket. Hot buckets are buckets that actively being written to. An index can have several hot buckets open at a time. Hot buckets are also searchable.

When certain conditions are met (for example, the hot bucket reaches a certain size or the indexer gets restarted), the hot bucket becomes a warm bucket ("rolls to warm"), and a new hot bucket is created in its place. The warm bucket is renamed but it remains in the same location as when it was a hot bucket. Warm buckets are searchable, but they are not actively written to. There can be a large number of warm buckets.

Once further conditions are met (for example, the index reaches some maximum number of warm buckets), the indexer begins to roll the warm buckets to cold, based on their age. It always selects the oldest warm bucket to roll to cold. Buckets continue to roll to cold as they age in this manner. Cold buckets reside in a different location from hot and warm buckets. You can configure the location so that cold buckets reside on cheaper storage.

Finally, after certain other time-based or size-based conditions are met, cold buckets roll to the frozen state, at which point they are deleted from the index, after being optionally archived.

If the frozen data has been archived, it can later be thawed. Data in thawed buckets is available for searches.

Settings in indexes.conf determine when a bucket moves from one state to the next.

They talk about 5 buckets and assuming that they already come with some default size(the max size per bucket) it could be 5 gb . answer#1.
It does also say that frozen buckets are deleted by default unless they are archived so I am not all sure!
I am just perhaps trying to force fit an answer given the options that you mentioned...
or could it be a typo in the question from the exam itself!!?
Sorry, if you had already considered these options and decided that they were incorrect.

gcusello
SplunkTrust
SplunkTrust

Hi Sukisen1981,
Thank you very much for your esteem.
Your interpretation could be correct and your answer only confirms what I thought: there isn't a precise point in which documentation says exactly what is the default value for the _internal index!

Thank you.
Giuseppe

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi Sukisen1981,
studying for the exam, I found this information at https://docs.splunk.com/Documentation/Splunk/7.3.1/Troubleshooting/WhatSplunklogsaboutitself

If you have any long-running real-time searches, you might want to adjust the maximum size of your search logs. These logs are rotated when they reach a default maximum size of 10 MB. Splunk software keeps up to five of them for each search, so the total log size for a search can conceivably grow as large as 30 MB.

do you think that this could be the exact answer?

Bye.
Giuseppe

0 Karma

Sukisen1981
Champion

this certainly matches the answer options, but I am not feeling sure, the section you cited begins with
The Splunk platform also creates search logs. These are not indexed to _internal.
So, is this specific to _internal index?
Wondering if someone from splunk in the forum can clear this for once and for all?

0 Karma

Sukisen1981
Champion

looks like it @gcusello ...but I agree with @woodcock I understand that there will be some straight knowledge questions , but this one is..well just too much 'nitty-gritty'
I have to re take both power user n adming by oct 19. I am thinking of taking power user first, have you guys taken the re-certi for that? If yes, any suggestions - atm i am just thinking of going thru fundamentals one and lots of old notes on fundamental 2 from the previous course, chiefly on macros, lookups.
I am feeling confident sometimes and sometimes not so much!
I certainly do not intend this to be the first IT certi exam I fail, but there is always that nagging doubt...

0 Karma

gcusello
SplunkTrust
SplunkTrust

Passed!
Thank you.
Bye.
Giuseppe

Sukisen1981
Champion

@gcusello
Any tips , I am going in for core power user (or whatever the power user exam is called now) on 14th Sep....

0 Karma

gcusello
SplunkTrust
SplunkTrust

check in details the exam program and be sure that you didn't forget any part of it!
good luck to the wolf!
bye.
Giuseppe

0 Karma

woodcock
Esteemed Legend

It must be but again, it is a silly thing to test about.

0 Karma
Get Updates on the Splunk Community!

Earn a $35 Gift Card for Answering our Splunk Admins & App Developer Survey

Survey for Splunk Admins and App Developers is open now! | Earn a $35 gift card!      Hello there,  Splunk ...

Continuing Innovation & New Integrations Unlock Full Stack Observability For Your ...

You’ve probably heard the latest about AppDynamics joining the Splunk Observability portfolio, deepening our ...

Monitoring Amazon Elastic Kubernetes Service (EKS)

As we’ve seen, integrating Kubernetes environments with Splunk Observability Cloud is a quick and easy way to ...