Index showing latest data as over a month ago but events are still coming in

[lkm93]

Hi,

I have recently started looking at .conf files and configuring them to log specific site data.

After I made my changes and everything was getting logged I have come across an odd issue where by my index is showing latest and earliest data is 20+ days ago. When I run searches and queries against the index there is data still coming in in real time. Anyone come across this and how would i go about trouble shooting this?

Example of my index:

IndexA  Earliest 23 days ago    Latest 23 days ago  $SPLUNK_DB/IndexA/db    

(Disclaimer I'm relatively new to this Splunk world please bear with me)

[lkm93]

Hi @nickhillscpl,

Here's the an attachment of the screenshot I took earlier this morning:

And below is a screenshot of the results take from the command you provided:

In the Splunk server column you will see server xxxxx_01. We have 3 index servers running on there as a cluster. the rest wont show the screenshot but i hope this makes the set up a bit clearer?

[nickhills]

And is xxxx_01 the server you are looking at in the first screenshot you posted in the question?

I may have guessed wrong wrong, but I wonder if you are looking at the index stats (where it says 23 days ago) on the search head, if so, then that is expected.

A SearchHead can't actually reflect the index statistics of your indexers.
Seeing as you have 3 indexers, have you tried looking at the index stats on xxxxx_01/2/3 - my instict is that the 3 indexers will have events from "a few seconds ago"
This is totally normal.

Arguably, you shouldnt have any user defined indexes on a SH - although it makes some management easier if you have a dummy index created on a SH with a very small max size. if you look at the index list on a SH ideally it should have NO events. The fact that you have a few thousand suggests something got miss allocated when you added the data to the platform.

[nickhills]

Are you sure they are being written to that index on that host? It looks very small and has less than a few thousand events?

[lkm93]

Hi @nickhillscpl ,

Yes that is odd to me too but I can confirm I still have events coming in in real-time. There are charts that depend on this index that are still displaying data as normal. some are real time

This wont let me upload a screenshot

[nickhills]

yeah answers is a bit annoying like that - post an answer, then you should be able to add a screenshot (after that you can convert your answer to a comment)

You dont state what type of server this is - is it a search head or an indexer?
Are you sure the events are not actually being written to a different indexer, albeit with the same index name?

Run a search like this:
<your events for this index>| eval indextime=strftime(_indextime,"%Y-%m-%d %H:%M:%S")|table _time indextime index splunk_server