Indexes + data retirement + Earliest Event

skippylou — Thu, 05 Aug 2010 18:47:16 GMT

So looking at the Indexes page in Manager, I can tell that one of my indexes has hit the size limit and is successfully retiring/deleting data as necessary to stay under the size limit I set for it. However, the 'Earliest Event' timestamp listed is for the earliest that has ever been in the index, not what is actually in the index currently.

Any (easy) way to have this always show the actual 'Earliest Event' based on what is actually in the index at that time?

Thanks,

Scott

Re: Indexes + data retirement + Earliest Event

Stephen_Sorkin — Sun, 15 Aug 2010 02:28:33 GMT

Excellent question.

This is because of the process for updating the metadata when a bucket is retired. It's relatively easy to decrement the counts for sources, sourcetypes and hosts when a bucket goes away. However, it's not efficient to update the earliest and latest timestamps, which are used for this display.

You have a couple of choices to fix this. You can either get at the data a different way, by means of the dbinspect command, or you could update the global {Hosts,Sources,SourceTypes}.data.

To retrieve the accurate time bounds with dbinspect, run the search:

| dbinspect index=<index_name> | convert timeformat="%m/%d/%Y:%T" mktime(earliestTime) mktime(latestTime) | stats min(earliestTime) as earliestTime max(latestTime) as latestTime | convert ctime(earliestTime) ctime(latestTime)

To update the metadata itself, you can create a meta.dirty file to cause the metadata to be regenerated:

touch $SPLUNK_HOME/var/lib/splunk/<index_name>/db/meta.dirty

topic Re: Indexes + data retirement + Earliest Event in Getting Data In

Indexes + data retirement + Earliest Event

Re: Indexes + data retirement + Earliest Event