When does metadata age out?

[kearaspoor]

I have a search that is looking through the metadata for high-priority/hig-volume hosts and it sucessfully alerts us if one of them hasn't sent in events in the previous 15 minutes.

| metadata type=hosts | search host=****| fields host, lastTime, recentTime | eval LastTime=strftime(lastTime, "%m-%d-%Y %H:%M:%S") | eval RecentTime=strftime(recentTime, "%m-%d-%Y %H:%M:%S") | eval dif=(now()-recentTime) | where dif>900 | rename dif AS eventLastRecieved_SecondsAgo | fields host RecentTime eventLastRecieved_SecondsAgo

The problem started showing up when one of these hosts was recently replaced and a rename occurred. For a short period of time, a server with a temporary name that matches the convention above was seen sending logs, as expected. When that server was renamed to the original server's name we started receiving alerts that the temporary name was no longer sending logs, as we'd expect. The problem is that we're still receiving those alerts several days later.

I've tried adjusting the timespan of the metadata search to only look at the metadata in the previous couple hours but the temporary hostname is still showing up.

So... How does metadata age? Will this host ever stop showing up? Or should I re-work the alert to show events where the last event was older than 15 minutes but less than X days?

[emiller42 · Accepted solution]

metadata age is directly tied to index age. When you run the metadata command you're basically looking at the indexed metadata pulled by splunk.

So if the old host name shows up in any index bucket, it'll show up in the metadata search. When you narrow the time range, it will return results from any index bucket that has an event within the specified time range. Again, if the old host is in any of those buckets, it'll return in the search.

So it'll eventually age, but the speed of that depends on your indexing volume and bucket sizing.

[kearaspoor]

I was suspecting something like that. Thanks for clearing it up!