I have recently started looking at .conf files and configuring them to log specific site data.
After I made my changes and everything was getting logged I have come across an odd issue where by my index is showing latest and earliest data is 20+ days ago. When I run searches and queries against the index there is data still coming in in real time. Anyone come across this and how would i go about trouble shooting this?
Example of my index:
IndexA Earliest 23 days ago Latest 23 days ago $SPLUNK_DB/IndexA/db
(Disclaimer I'm relatively new to this Splunk world please bear with me)
Here's the an attachment of the screenshot I took earlier this morning:
And below is a screenshot of the results take from the command you provided:
In the Splunk server column you will see server xxxxx_01. We have 3 index servers running on there as a cluster. the rest wont show the screenshot but i hope this makes the set up a bit clearer?
And is xxxx_01 the server you are looking at in the first screenshot you posted in the question?
I may have guessed wrong wrong, but I wonder if you are looking at the index stats (where it says 23 days ago) on the search head, if so, then that is expected.
A SearchHead can't actually reflect the index statistics of your indexers.
Seeing as you have 3 indexers, have you tried looking at the index stats on xxxxx_01/2/3 - my instict is that the 3 indexers will have events from "a few seconds ago"
This is totally normal.
Arguably, you shouldnt have any user defined indexes on a SH - although it makes some management easier if you have a dummy index created on a SH with a very small max size. if you look at the index list on a SH ideally it should have NO events. The fact that you have a few thousand suggests something got miss allocated when you added the data to the platform.
Hi @nickhillscpl ,
Yes that is odd to me too but I can confirm I still have events coming in in real-time. There are charts that depend on this index that are still displaying data as normal. some are real time
This wont let me upload a screenshot
yeah answers is a bit annoying like that - post an answer, then you should be able to add a screenshot (after that you can convert your answer to a comment)
You dont state what type of server this is - is it a search head or an indexer?
Are you sure the events are not actually being written to a different indexer, albeit with the same index name?
Run a search like this:
<your events for this index>| eval indextime=strftime(_indextime,"%Y-%m-%d %H:%M:%S")|table _time indextime index splunk_server