Getting Data In

How to see historical metrics in splunkd.log and metrics.log on my indexers?

vw5qb73
Explorer

I want to see historical metrics in splunkd.log/metrics.log on my indexers. Currently i see only 1 days data.

Is there any setting that is controlling this behaviour?

0 Karma

Richfez
SplunkTrust
SplunkTrust

The retention of your indexes for _internal will determine this, but it's set far higher than one day by default so you should be OK at least to 30 days and probably for far more.

So, if you run a search like

index=_internal

Over various time periods (last 7 days, last 30 days, etc...) you can find out how far back you have logs being saved at this time.

Where exactly are you seeing one day (and one day only) of data? For most of the searches I know of, you can mouse-over the search/report in the dashboard and in the lower left (6.4 and earlier) or lower right (6.5+) there's a magnifying glass you can click to "Open in search", which at that point you can change the date/time ranges to anything you'd like.

Also on many pages there's a drop-down that changes the date/time range.

Give those a try and if they're not working for you, please give us more information about exactly where you see only a single day of data?

0 Karma

vw5qb73
Explorer

When i try this i am seeing only 1 day's events not before that for all source's and sourcetypes

0 Karma

Richfez
SplunkTrust
SplunkTrust

Assuming these are your indexers (if you aren't the admin, there's some trickery with search filtering the admins could be doing) ...

First, what exactly do you mean by "1 day's events?" Do events go back 24 hours (exactly? rounded to nearest hour?) or do you see all events after, say, midnight last night but nothing from 11:59 and earlier? It wasn't terribly important before, but now we'll have to dig a little deeper so we need to know.

Go to Settings, Indexes. Look down that list (especially for _internal) and see waht it says. What's the earliest event date? What's the latest? Do they show something like you expect, or only the last day or two?

So, if your indexes only show they have recent data, then it's most likely there's a setting on them doing this. Click on _internal (in Indexes) and make sure it's set big enough.

If the indexes say they have data going back several days or more, then we'll have to look down possible search issues. We'll cross that bridge when we come to it.

0 Karma

vw5qb73
Explorer

I am referring to indexers. Events go back only till last 24hours not beyond that, this makes me think there is some setting that is controlling this behaviour

0 Karma

Richfez
SplunkTrust
SplunkTrust

Right, that's what I was figuring and hoping.

Did you check the settings on your indexes? There are settings in there that determine a lot about how long retention will be. These are very likely to be the culprit.

Specifically, on _internal what does Settings/Indexes say is the oldest event date and the latest? What size is it? What is the max size set for it? And let us know the same set of information about one other index you are having difficulties with.

On any index, if the maximum size is set to X and the current size also shows X and oldest event is not long enough ago then the likely answer is to simply increase the size of the index. If that simple solution doesn't work, please let us know what those settings and values are so we can figure out how to proceed.

IF (and only if) the above doesn't resolve this problem, then please get a little more specific with what you mean by "last 24 hours". Is it precise to the minute? Like right now at 11:49 AM (or whatever time it is there) what's the oldest data you see when you do a simple index=* (or index=_internal) search over all time? 11:49 AM yesterday? Noon yesterday? 11:00 AM yesterday? If you keep running that above search every few minutes, does it stay "oldest as 11:49 AM" for like 15 minutes then all of a sudden the oldest event is now 12:17 yesterday, or does it smoothly just stay 24 hours ago (to the minute/second, or just nearest hour?) All of these could potentially be indicators of different issues/problems.

And Merry Christmas!

Happy Splunking,
Rich

0 Karma

Richfez
SplunkTrust
SplunkTrust

Just checking in - have you resolved this issue (or perhaps it resolved itself?) Do you need additional help?

0 Karma
Register for .conf21 Now! Go Vegas or Go Virtual!

How will you .conf21? You decide! Go in-person in Las Vegas, 10/18-10/21, or go online with .conf21 Virtual, 10/19-10/20.