Getting Data In

Index showing latest data as over a month ago but events are still coming in

lkm93
Explorer

Hi,

I have recently started looking at .conf files and configuring them to log specific site data.

After I made my changes and everything was getting logged I have come across an odd issue where by my index is showing latest and earliest data is 20+ days ago. When I run searches and queries against the index there is data still coming in in real time. Anyone come across this and how would i go about trouble shooting this?

Example of my index:

alt text

IndexA  Earliest 23 days ago    Latest 23 days ago  $SPLUNK_DB/IndexA/db    

(Disclaimer I'm relatively new to this Splunk world please bear with me)

0 Karma

lkm93
Explorer

Hi @nickhillscpl,

Here's the an attachment of the screenshot I took earlier this morning:

alt text

And below is a screenshot of the results take from the command you provided:

alt text

In the Splunk server column you will see server xxxxx_01. We have 3 index servers running on there as a cluster. the rest wont show the screenshot but i hope this makes the set up a bit clearer?

0 Karma

nickhills
Ultra Champion

And is xxxx_01 the server you are looking at in the first screenshot you posted in the question?

I may have guessed wrong wrong, but I wonder if you are looking at the index stats (where it says 23 days ago) on the search head, if so, then that is expected.

A SearchHead can't actually reflect the index statistics of your indexers.
Seeing as you have 3 indexers, have you tried looking at the index stats on xxxxx_01/2/3 - my instict is that the 3 indexers will have events from "a few seconds ago"
This is totally normal.

Arguably, you shouldnt have any user defined indexes on a SH - although it makes some management easier if you have a dummy index created on a SH with a very small max size. if you look at the index list on a SH ideally it should have NO events. The fact that you have a few thousand suggests something got miss allocated when you added the data to the platform.

If my comment helps, please give it a thumbs up!
0 Karma

nickhills
Ultra Champion

Are you sure they are being written to that index on that host? It looks very small and has less than a few thousand events?

If my comment helps, please give it a thumbs up!
0 Karma

lkm93
Explorer

Hi @nickhillscpl ,

Yes that is odd to me too but I can confirm I still have events coming in in real-time. There are charts that depend on this index that are still displaying data as normal. some are real time

This wont let me upload a screenshot

0 Karma

nickhills
Ultra Champion

yeah answers is a bit annoying like that - post an answer, then you should be able to add a screenshot (after that you can convert your answer to a comment)

You dont state what type of server this is - is it a search head or an indexer?
Are you sure the events are not actually being written to a different indexer, albeit with the same index name?

Run a search like this:
<your events for this index>| eval indextime=strftime(_indextime,"%Y-%m-%d %H:%M:%S")|table _time indextime index splunk_server

If my comment helps, please give it a thumbs up!
0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...